2023-07-14T18:54:28.313 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e 2023-07-14T18:54:28.314 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --verbose --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} --name pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051 -- /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/orig.config.yaml 2023-07-14T18:54:28.379 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T18:54:33.099 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051 branch: wip-pdonnell-testing-20230714.134335 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} email: pdonnell@redhat.com first_in_suite: false job_id: '7338051' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230714.134335 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230714.134335 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5179 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec sleep_before_teardown: 0 subset: 285/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230714.134335 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi053.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi070.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs _nogroup distributed 1 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - kernel_untar_build.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: aa6a80e6cc655fff48452a7eca02070e61d074fc timestamp: 2023-07-14_16:44:47 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-14T18:54:33.100 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa; will attempt to use it 2023-07-14T18:54:33.101 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa/tasks 2023-07-14T18:54:33.101 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-14T18:54:33.103 INFO:teuthology.task.internal:Checking packages... 2023-07-14T18:54:33.131 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '81c9387d2e7576da065747ec9c16d1c7d69016ec' 2023-07-14T18:54:33.132 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-14T18:54:33.132 INFO:teuthology.packaging:ref: None 2023-07-14T18:54:33.132 INFO:teuthology.packaging:tag: None 2023-07-14T18:54:33.133 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230714.134335 2023-07-14T18:54:33.133 INFO:teuthology.packaging:sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:54:33.133 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230714.134335 2023-07-14T18:54:43.408 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4940.g81c9387d 2023-07-14T18:54:43.409 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-14T18:54:43.415 INFO:teuthology.task.internal:no buildpackages task found 2023-07-14T18:54:43.415 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-14T18:54:43.423 INFO:teuthology.task.internal:Saving configuration 2023-07-14T18:54:43.449 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-14T18:54:43.456 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:54:43.486 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:49:07.309276', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:54:43.512 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:49:07.307930', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:54:43.539 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:49:07.306370', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:54:43.539 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-14T18:54:43.546 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-14T18:54:43.546 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-14T18:54:43.546 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-14T18:54:43.547 INFO:teuthology.run_tasks:Running task console_log... 2023-07-14T18:54:43.664 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb7bdd90430>, signals=[15]) 2023-07-14T18:54:43.666 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-14T18:54:43.672 INFO:teuthology.task.internal:Opening connections... 2023-07-14T18:54:43.672 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2023-07-14T18:54:43.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:54:43.746 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2023-07-14T18:54:43.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:54:43.816 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2023-07-14T18:54:43.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:54:43.886 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-14T18:54:43.893 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2023-07-14T18:54:43.914 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2023-07-14T18:54:43.914 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2023-07-14T18:54:43.973 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:54:43.973 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:54:43.973 INFO:teuthology.orchestra.run.smithi053.stdout:ID="rhel" 2023-07-14T18:54:43.974 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="fedora" 2023-07-14T18:54:43.974 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="8.6" 2023-07-14T18:54:43.974 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:54:43.974 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:54:43.974 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2023-07-14T18:54:43.974 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:54:43.974 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:54:43.975 INFO:teuthology.orchestra.run.smithi053.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:54:43.975 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:54:43.975 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T18:54:43.975 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:54:43.975 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:54:43.975 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:54:43.975 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:54:43.976 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2023-07-14T18:54:43.999 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2023-07-14T18:54:44.019 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2023-07-14T18:54:44.019 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2023-07-14T18:54:44.076 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:54:44.077 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:54:44.077 INFO:teuthology.orchestra.run.smithi070.stdout:ID="rhel" 2023-07-14T18:54:44.077 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="fedora" 2023-07-14T18:54:44.078 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="8.6" 2023-07-14T18:54:44.078 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:54:44.078 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:54:44.079 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2023-07-14T18:54:44.079 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:54:44.079 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:54:44.080 INFO:teuthology.orchestra.run.smithi070.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:54:44.080 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:54:44.081 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T18:54:44.081 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:54:44.081 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:54:44.081 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:54:44.082 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:54:44.083 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2023-07-14T18:54:49.114 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2023-07-14T18:54:49.149 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2023-07-14T18:54:49.150 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2023-07-14T18:54:49.207 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:54:49.208 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:54:49.208 INFO:teuthology.orchestra.run.smithi104.stdout:ID="rhel" 2023-07-14T18:54:49.208 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="fedora" 2023-07-14T18:54:49.208 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8.6" 2023-07-14T18:54:49.208 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:54:49.209 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:54:49.209 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2023-07-14T18:54:49.209 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:54:49.209 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:54:49.209 INFO:teuthology.orchestra.run.smithi104.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:54:49.209 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:54:49.209 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T18:54:49.210 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:54:49.210 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:54:49.210 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:54:49.210 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:54:49.210 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2023-07-14T18:54:54.239 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-14T18:54:54.251 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-14T18:54:54.270 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-14T18:54:54.270 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:54:54.273 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:54:54.275 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:54:54.294 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-14T18:54:54.302 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-14T18:54:54.302 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:54:54.333 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:54:54.336 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:54:54.367 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-14T18:55:04.294 INFO:teuthology.run_tasks:Running task kernel... 2023-07-14T18:55:04.310 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-07-14T18:55:04.310 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-07-14T18:55:04.311 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-07-14T18:55:04.311 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:55:04.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:55:04.332 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2023-07-14T18:55:04.391 INFO:teuthology.orchestra.run.smithi053.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T18:55:04.392 INFO:teuthology.task.kernel:Running kernel on smithi053: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:55:04.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2023-07-14T18:55:05.615 INFO:teuthology.orchestra.run.smithi053.stdout:Updating Subscription Management repositories. 2023-07-14T18:55:05.616 INFO:teuthology.orchestra.run.smithi053.stdout:Unable to read consumer identity 2023-07-14T18:55:07.515 INFO:teuthology.orchestra.run.smithi053.stdout:Copr repo for python3-asyncssh owned by ceph 5.5 kB/s | 3.5 kB 00:00 2023-07-14T18:55:13.598 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 16 MB 00:06 2023-07-14T18:55:17.256 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 428 kB/s | 24 kB 00:00 2023-07-14T18:55:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T18:55:18.026 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-07-14T18:55:18.027 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-07-14T18:55:18.028 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-07-14T18:55:18.144 DEBUG:teuthology.orchestra.run.smithi053:> echo no | sudo yum reinstall kernel || true 2023-07-14T18:55:18.494 INFO:teuthology.orchestra.run.smithi053.stdout:Updating Subscription Management repositories. 2023-07-14T18:55:18.494 INFO:teuthology.orchestra.run.smithi053.stdout:Unable to read consumer identity 2023-07-14T18:55:18.777 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:01 ago on Fri 14 Jul 2023 06:55:17 PM UTC. 2023-07-14T18:55:18.818 INFO:teuthology.orchestra.run.smithi053.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:55:18.852 INFO:teuthology.orchestra.run.smithi053.stderr:Error: No packages marked for reinstall. 2023-07-14T18:55:18.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum reinstall -y kernel || true 2023-07-14T18:55:19.248 INFO:teuthology.orchestra.run.smithi053.stdout:Updating Subscription Management repositories. 2023-07-14T18:55:19.249 INFO:teuthology.orchestra.run.smithi053.stdout:Unable to read consumer identity 2023-07-14T18:55:19.514 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 06:55:17 PM UTC. 2023-07-14T18:55:19.557 INFO:teuthology.orchestra.run.smithi053.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:55:19.576 INFO:teuthology.orchestra.run.smithi053.stderr:Error: No packages marked for reinstall. 2023-07-14T18:55:19.623 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T18:55:19.706 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T18:55:19.706 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:55:19.706 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T18:55:19.706 DEBUG:teuthology.parallel:result is None 2023-07-14T18:55:19.707 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-14T18:55:19.718 INFO:teuthology.task.internal:Creating test directory... 2023-07-14T18:55:19.719 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T18:55:19.722 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T18:55:19.724 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T18:55:19.745 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-14T18:55:19.754 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-14T18:55:19.762 INFO:teuthology.task.internal:Creating archive directory... 2023-07-14T18:55:19.762 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T18:55:19.778 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T18:55:19.785 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T18:55:19.828 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-14T18:55:19.835 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-14T18:55:19.835 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:55:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:55:19.877 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:55:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:55:19.895 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T18:55:19.897 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T18:55:19.919 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T18:55:19.929 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:55:19.940 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:55:19.957 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:55:19.991 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:55:19.996 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:55:20.014 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:55:20.016 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-14T18:55:20.023 INFO:teuthology.task.internal:Configuring sudo... 2023-07-14T18:55:20.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T18:55:20.025 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T18:55:20.040 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T18:55:20.068 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-14T18:55:20.081 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-14T18:55:20.081 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T18:55:20.094 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T18:55:20.109 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T18:55:20.126 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:55:20.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:55:20.559 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:55:20.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:55:20.685 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T18:55:20.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T18:55:20.753 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:55:25.807 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:55:25.850 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:55:30.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:55:30.972 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T18:55:30.972 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T18:55:31.261 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:55:36.317 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:55:36.354 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:55:39.773 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:55:39.805 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T18:55:39.805 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T18:55:39.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2023-07-14T18:55:39.879 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2023-07-14T18:55:39.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2023-07-14T18:55:39.962 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:55:39.968 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:55:39.982 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:55:40.339 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-14T18:55:40.347 INFO:teuthology.task.internal:Starting timer... 2023-07-14T18:55:40.347 INFO:teuthology.run_tasks:Running task pcp... 2023-07-14T18:55:40.362 INFO:teuthology.run_tasks:Running task selinux... 2023-07-14T18:55:40.371 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-07-14T18:55:40.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2023-07-14T18:55:40.439 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2023-07-14T18:55:40.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2023-07-14T18:55:40.508 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2023-07-14T18:55:40.510 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2023-07-14T18:55:40.574 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2023-07-14T18:55:40.576 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-14T18:55:40.576 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2023-07-14T18:55:40.596 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2023-07-14T18:55:40.596 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2023-07-14T18:55:40.626 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2023-07-14T18:55:40.627 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2023-07-14T18:55:40.648 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2023-07-14T18:55:40.648 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive'} 2023-07-14T18:55:40.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:55:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:55:40.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:55:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:55:40.706 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:55:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:55:40.736 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-14T18:55:40.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2023-07-14T18:55:40.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2023-07-14T18:55:40.796 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2023-07-14T18:55:40.824 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-14T18:55:40.833 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-14T18:55:40.839 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-14T18:55:40.971 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-14T18:55:40.989 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-14T18:55:40.991 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com,smithi070.front.sepia.ceph.com,smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-14T19:05:30.488 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2023-07-14T19:05:30.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-07-14T19:05:30.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T19:05:35.567 DEBUG:teuthology.orchestra.run.smithi053:> true 2023-07-14T19:05:35.646 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-07-14T19:05:35.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2023-07-14T19:05:35.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T19:05:40.724 DEBUG:teuthology.orchestra.run.smithi070:> true 2023-07-14T19:05:40.796 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2023-07-14T19:05:40.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2023-07-14T19:05:40.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T19:05:45.871 DEBUG:teuthology.orchestra.run.smithi104:> true 2023-07-14T19:05:45.944 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2023-07-14T19:05:45.945 INFO:teuthology.run_tasks:Running task clock... 2023-07-14T19:05:45.955 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-14T19:05:45.956 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T19:05:45.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:05:45.959 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T19:05:45.960 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:05:45.962 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T19:05:45.962 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:05:45.996 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T19:05:45.997 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T19:05:46.012 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T19:05:46.013 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T19:05:46.020 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T19:05:46.037 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T19:05:46.089 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2023-07-14T19:05:46.090 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2023-07-14T19:05:46.101 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2023-07-14T19:05:46.103 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2023-07-14T19:05:46.106 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2023-07-14T19:05:46.116 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T19:05:46.118 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2023-07-14T19:05:46.119 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T19:05:46.130 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T19:05:46.133 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T19:05:46.133 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T19:05:46.148 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T19:05:46.209 INFO:teuthology.orchestra.run.smithi070.stderr:bash: ntpq: command not found 2023-07-14T19:05:46.213 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:05:46.213 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2023-07-14T19:05:46.213 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.214 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.214 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.214 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.216 INFO:teuthology.orchestra.run.smithi053.stderr:bash: ntpq: command not found 2023-07-14T19:05:46.220 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:05:46.220 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2023-07-14T19:05:46.221 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.221 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.221 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.222 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.235 INFO:teuthology.orchestra.run.smithi104.stderr:bash: ntpq: command not found 2023-07-14T19:05:46.239 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:05:46.239 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2023-07-14T19:05:46.239 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.239 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.239 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.240 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:05:46.240 INFO:teuthology.run_tasks:Running task pexec... 2023-07-14T19:05:46.252 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-14T19:05:46.253 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi053.front.sepia.ceph.com 2023-07-14T19:05:46.254 DEBUG:teuthology.orchestra.run.smithi053:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T19:05:46.254 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi070.front.sepia.ceph.com 2023-07-14T19:05:46.255 DEBUG:teuthology.orchestra.run.smithi070:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T19:05:46.256 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi104.front.sepia.ceph.com 2023-07-14T19:05:46.256 DEBUG:teuthology.orchestra.run.smithi104:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T19:05:46.627 INFO:teuthology.orchestra.run.smithi070.stdout:Updating Subscription Management repositories. 2023-07-14T19:05:46.651 INFO:teuthology.orchestra.run.smithi053.stdout:Updating Subscription Management repositories. 2023-07-14T19:05:46.652 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2023-07-14T19:05:48.222 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:00 ago on Fri 14 Jul 2023 07:03:47 PM UTC. 2023-07-14T19:05:48.250 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:46 ago on Fri 14 Jul 2023 07:03:01 PM UTC. 2023-07-14T19:05:48.303 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:46 ago on Fri 14 Jul 2023 07:03:01 PM UTC. 2023-07-14T19:05:49.986 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2023-07-14T19:05:49.987 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2023-07-14T19:05:49.987 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2023-07-14T19:05:49.988 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2023-07-14T19:05:49.988 INFO:teuthology.orchestra.run.smithi104.stdout:Resetting modules: 2023-07-14T19:05:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: container-tools 2023-07-14T19:05:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:05:49.988 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2023-07-14T19:05:49.988 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2023-07-14T19:05:49.988 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:05:50.180 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-07-14T19:05:50.181 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-07-14T19:05:50.181 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2023-07-14T19:05:50.181 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-07-14T19:05:50.181 INFO:teuthology.orchestra.run.smithi070.stdout:Resetting modules: 2023-07-14T19:05:50.181 INFO:teuthology.orchestra.run.smithi070.stdout: container-tools 2023-07-14T19:05:50.182 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:05:50.182 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-07-14T19:05:50.182 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2023-07-14T19:05:50.182 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:05:50.220 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-07-14T19:05:50.221 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-07-14T19:05:50.222 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2023-07-14T19:05:50.222 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-07-14T19:05:50.223 INFO:teuthology.orchestra.run.smithi053.stdout:Resetting modules: 2023-07-14T19:05:50.223 INFO:teuthology.orchestra.run.smithi053.stdout: container-tools 2023-07-14T19:05:50.223 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:05:50.223 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-07-14T19:05:50.223 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-07-14T19:05:50.223 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:05:50.958 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:49 ago on Fri 14 Jul 2023 07:03:01 PM UTC. 2023-07-14T19:05:51.209 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:03 ago on Fri 14 Jul 2023 07:03:47 PM UTC. 2023-07-14T19:05:51.268 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:49 ago on Fri 14 Jul 2023 07:03:01 PM UTC. 2023-07-14T19:05:58.559 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-07-14T19:05:58.877 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-07-14T19:05:59.032 INFO:teuthology.orchestra.run.smithi070.stdout:Updating Subscription Management repositories. 2023-07-14T19:05:59.241 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2023-07-14T19:05:59.354 INFO:teuthology.orchestra.run.smithi053.stdout:Updating Subscription Management repositories. 2023-07-14T19:05:59.736 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2023-07-14T19:06:00.610 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:59 ago on Fri 14 Jul 2023 07:03:01 PM UTC. 2023-07-14T19:06:01.154 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:13 ago on Fri 14 Jul 2023 07:03:47 PM UTC. 2023-07-14T19:06:01.326 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:59 ago on Fri 14 Jul 2023 07:03:01 PM UTC. 2023-07-14T19:06:02.468 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-07-14T19:06:02.561 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================================ 2023-07-14T19:06:02.562 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-07-14T19:06:02.562 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================================ 2023-07-14T19:06:02.562 INFO:teuthology.orchestra.run.smithi070.stdout:Installing group/module packages: 2023-07-14T19:06:02.562 INFO:teuthology.orchestra.run.smithi070.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T19:06:02.562 INFO:teuthology.orchestra.run.smithi070.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T19:06:02.562 INFO:teuthology.orchestra.run.smithi070.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T19:06:02.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T19:06:02.563 INFO:teuthology.orchestra.run.smithi070.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T19:06:02.563 INFO:teuthology.orchestra.run.smithi070.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T19:06:02.563 INFO:teuthology.orchestra.run.smithi070.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T19:06:02.563 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-07-14T19:06:02.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T19:06:02.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T19:06:02.564 INFO:teuthology.orchestra.run.smithi070.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T19:06:02.564 INFO:teuthology.orchestra.run.smithi070.stdout:Installing module profiles: 2023-07-14T19:06:02.565 INFO:teuthology.orchestra.run.smithi070.stdout: container-tools/common 2023-07-14T19:06:02.565 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:06:02.565 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling module streams: 2023-07-14T19:06:02.565 INFO:teuthology.orchestra.run.smithi070.stdout: container-tools rhel8 2023-07-14T19:06:02.565 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:06:02.566 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-07-14T19:06:02.566 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================================ 2023-07-14T19:06:02.566 INFO:teuthology.orchestra.run.smithi070.stdout:Install 10 Packages 2023-07-14T19:06:02.566 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:06:02.567 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 18 M 2023-07-14T19:06:02.567 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 65 M 2023-07-14T19:06:02.567 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-07-14T19:06:02.802 INFO:teuthology.orchestra.run.smithi070.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 392 kB/s | 94 kB 00:00 2023-07-14T19:06:02.828 INFO:teuthology.orchestra.run.smithi070.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 154 kB/s | 41 kB 00:00 2023-07-14T19:06:02.903 INFO:teuthology.orchestra.run.smithi070.stdout:(3/10): udica-0.2.6-4.module+el8.6.0+16771+28df 492 kB/s | 49 kB 00:00 2023-07-14T19:06:02.920 INFO:teuthology.orchestra.run.smithi070.stdout:(4/10): python3-pytoml-0.1.14-5.git7dea353.el8. 69 kB/s | 25 kB 00:00 2023-07-14T19:06:02.995 INFO:teuthology.orchestra.run.smithi070.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.1 MB/s | 524 kB 00:00 2023-07-14T19:06:03.036 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2023-07-14T19:06:03.062 INFO:teuthology.orchestra.run.smithi070.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.0 MB/s | 149 kB 00:00 2023-07-14T19:06:03.129 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================================ 2023-07-14T19:06:03.129 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2023-07-14T19:06:03.129 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================================ 2023-07-14T19:06:03.129 INFO:teuthology.orchestra.run.smithi104.stdout:Installing group/module packages: 2023-07-14T19:06:03.129 INFO:teuthology.orchestra.run.smithi104.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T19:06:03.129 INFO:teuthology.orchestra.run.smithi104.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T19:06:03.130 INFO:teuthology.orchestra.run.smithi104.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T19:06:03.130 INFO:teuthology.orchestra.run.smithi104.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T19:06:03.130 INFO:teuthology.orchestra.run.smithi104.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T19:06:03.130 INFO:teuthology.orchestra.run.smithi104.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T19:06:03.130 INFO:teuthology.orchestra.run.smithi104.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T19:06:03.130 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2023-07-14T19:06:03.130 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T19:06:03.131 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T19:06:03.131 INFO:teuthology.orchestra.run.smithi104.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T19:06:03.131 INFO:teuthology.orchestra.run.smithi104.stdout:Installing module profiles: 2023-07-14T19:06:03.131 INFO:teuthology.orchestra.run.smithi104.stdout: container-tools/common 2023-07-14T19:06:03.131 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:06:03.131 INFO:teuthology.orchestra.run.smithi104.stdout:Enabling module streams: 2023-07-14T19:06:03.131 INFO:teuthology.orchestra.run.smithi104.stdout: container-tools rhel8 2023-07-14T19:06:03.131 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:06:03.132 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2023-07-14T19:06:03.132 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================================ 2023-07-14T19:06:03.132 INFO:teuthology.orchestra.run.smithi104.stdout:Install 10 Packages 2023-07-14T19:06:03.132 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:06:03.133 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 18 M 2023-07-14T19:06:03.133 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 65 M 2023-07-14T19:06:03.133 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2023-07-14T19:06:03.137 INFO:teuthology.orchestra.run.smithi070.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.5 MB/s | 213 kB 00:00 2023-07-14T19:06:03.158 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-07-14T19:06:03.238 INFO:teuthology.orchestra.run.smithi070.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 24 MB/s | 8.0 MB 00:00 2023-07-14T19:06:03.259 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================================ 2023-07-14T19:06:03.259 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-07-14T19:06:03.259 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================================ 2023-07-14T19:06:03.259 INFO:teuthology.orchestra.run.smithi053.stdout:Installing group/module packages: 2023-07-14T19:06:03.259 INFO:teuthology.orchestra.run.smithi053.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T19:06:03.259 INFO:teuthology.orchestra.run.smithi053.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T19:06:03.260 INFO:teuthology.orchestra.run.smithi053.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T19:06:03.260 INFO:teuthology.orchestra.run.smithi053.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T19:06:03.260 INFO:teuthology.orchestra.run.smithi053.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T19:06:03.260 INFO:teuthology.orchestra.run.smithi053.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T19:06:03.260 INFO:teuthology.orchestra.run.smithi053.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T19:06:03.260 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-07-14T19:06:03.260 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T19:06:03.261 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T19:06:03.261 INFO:teuthology.orchestra.run.smithi053.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T19:06:03.261 INFO:teuthology.orchestra.run.smithi053.stdout:Installing module profiles: 2023-07-14T19:06:03.261 INFO:teuthology.orchestra.run.smithi053.stdout: container-tools/common 2023-07-14T19:06:03.261 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:06:03.262 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling module streams: 2023-07-14T19:06:03.262 INFO:teuthology.orchestra.run.smithi053.stdout: container-tools rhel8 2023-07-14T19:06:03.262 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:06:03.262 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-07-14T19:06:03.262 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================================ 2023-07-14T19:06:03.262 INFO:teuthology.orchestra.run.smithi053.stdout:Install 10 Packages 2023-07-14T19:06:03.262 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:06:03.263 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 18 M 2023-07-14T19:06:03.263 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 65 M 2023-07-14T19:06:03.263 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-07-14T19:06:03.317 INFO:teuthology.orchestra.run.smithi104.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 507 kB/s | 94 kB 00:00 2023-07-14T19:06:03.339 INFO:teuthology.orchestra.run.smithi104.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 119 kB/s | 25 kB 00:00 2023-07-14T19:06:03.356 INFO:teuthology.orchestra.run.smithi104.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 181 kB/s | 41 kB 00:00 2023-07-14T19:06:03.380 INFO:teuthology.orchestra.run.smithi070.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 22 MB/s | 7.0 MB 00:00 2023-07-14T19:06:03.398 INFO:teuthology.orchestra.run.smithi104.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 607 kB/s | 49 kB 00:00 2023-07-14T19:06:03.401 INFO:teuthology.orchestra.run.smithi053.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 292 kB/s | 41 kB 00:00 2023-07-14T19:06:03.421 INFO:teuthology.orchestra.run.smithi070.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 7.7 MB/s | 2.2 MB 00:00 2023-07-14T19:06:03.422 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:06:03.422 INFO:teuthology.orchestra.run.smithi070.stdout:Total 21 MB/s | 18 MB 00:00 2023-07-14T19:06:03.443 INFO:teuthology.orchestra.run.smithi053.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 516 kB/s | 94 kB 00:00 2023-07-14T19:06:03.469 INFO:teuthology.orchestra.run.smithi053.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 120 kB/s | 25 kB 00:00 2023-07-14T19:06:03.473 INFO:teuthology.orchestra.run.smithi104.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.8 MB/s | 524 kB 00:00 2023-07-14T19:06:03.532 INFO:teuthology.orchestra.run.smithi104.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.1 MB/s | 149 kB 00:00 2023-07-14T19:06:03.536 INFO:teuthology.orchestra.run.smithi053.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 368 kB/s | 49 kB 00:00 2023-07-14T19:06:03.611 INFO:teuthology.orchestra.run.smithi053.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.1 MB/s | 524 kB 00:00 2023-07-14T19:06:03.622 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-07-14T19:06:03.632 INFO:teuthology.orchestra.run.smithi104.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 29 MB/s | 8.0 MB 00:00 2023-07-14T19:06:03.641 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-07-14T19:06:03.641 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-07-14T19:06:03.645 INFO:teuthology.orchestra.run.smithi053.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.3 MB/s | 149 kB 00:00 2023-07-14T19:06:03.658 INFO:teuthology.orchestra.run.smithi104.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.1 MB/s | 213 kB 00:00 2023-07-14T19:06:03.821 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-07-14T19:06:03.825 INFO:teuthology.orchestra.run.smithi104.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 24 MB/s | 7.0 MB 00:00 2023-07-14T19:06:03.826 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-07-14T19:06:03.828 INFO:teuthology.orchestra.run.smithi053.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 22 MB/s | 8.0 MB 00:00 2023-07-14T19:06:03.854 INFO:teuthology.orchestra.run.smithi053.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 879 kB/s | 213 kB 00:00 2023-07-14T19:06:03.902 INFO:teuthology.orchestra.run.smithi104.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 8.1 MB/s | 2.2 MB 00:00 2023-07-14T19:06:03.903 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:06:03.903 INFO:teuthology.orchestra.run.smithi104.stdout:Total 24 MB/s | 18 MB 00:00 2023-07-14T19:06:03.987 INFO:teuthology.orchestra.run.smithi053.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 21 MB/s | 7.0 MB 00:00 2023-07-14T19:06:04.054 INFO:teuthology.orchestra.run.smithi053.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 9.7 MB/s | 2.2 MB 00:00 2023-07-14T19:06:04.055 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:06:04.055 INFO:teuthology.orchestra.run.smithi053.stdout:Total 23 MB/s | 18 MB 00:00 2023-07-14T19:06:04.104 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2023-07-14T19:06:04.123 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2023-07-14T19:06:04.123 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2023-07-14T19:06:04.260 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-07-14T19:06:04.280 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-07-14T19:06:04.281 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-07-14T19:06:04.304 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2023-07-14T19:06:04.308 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2023-07-14T19:06:04.325 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-07-14T19:06:04.474 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-07-14T19:06:04.479 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-07-14T19:06:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T19:06:04.736 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:06:04.844 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2023-07-14T19:06:04.930 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T19:06:04.998 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-07-14T19:06:05.062 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T19:06:05.133 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T19:06:05.231 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T19:06:05.234 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:06:05.379 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:06:05.436 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T19:06:05.498 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T19:06:05.576 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T19:06:05.672 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T19:06:05.767 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T19:06:06.110 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T19:06:06.154 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T19:06:06.251 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T19:06:06.893 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T19:06:06.940 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T19:06:07.141 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T19:06:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T19:06:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T19:06:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:06:07.797 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T19:06:07.829 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T19:06:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:06:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T19:06:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:06:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T19:06:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T19:06:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T19:06:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T19:06:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T19:06:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T19:06:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T19:06:08.039 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T19:06:08.063 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T19:06:08.199 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T19:06:08.227 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T19:06:08.368 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:06:08.395 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:06:08.635 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:06:08.636 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T19:06:08.636 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:06:08.636 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T19:06:08.636 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T19:06:08.637 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T19:06:08.637 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T19:06:08.637 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T19:06:08.637 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T19:06:08.638 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T19:06:08.680 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:06:08.680 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T19:06:08.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:06:08.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T19:06:08.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T19:06:08.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T19:06:08.682 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T19:06:08.682 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T19:06:08.682 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T19:06:08.682 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T19:06:09.124 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T19:06:09.125 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:03:07 ago on Fri 14 Jul 2023 07:03:01 PM UTC. 2023-07-14T19:06:09.741 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T19:06:09.741 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:03:08 ago on Fri 14 Jul 2023 07:03:01 PM UTC. 2023-07-14T19:06:09.834 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T19:06:09.834 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:22 ago on Fri 14 Jul 2023 07:03:47 PM UTC. 2023-07-14T19:06:16.742 INFO:teuthology.orchestra.run.smithi070.stdout:Installed products updated. 2023-07-14T19:06:16.937 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:06:16.938 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-07-14T19:06:16.938 INFO:teuthology.orchestra.run.smithi070.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:16.938 INFO:teuthology.orchestra.run.smithi070.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:06:16.938 INFO:teuthology.orchestra.run.smithi070.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:16.939 INFO:teuthology.orchestra.run.smithi070.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T19:06:16.939 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T19:06:16.939 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T19:06:16.940 INFO:teuthology.orchestra.run.smithi070.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:16.940 INFO:teuthology.orchestra.run.smithi070.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:16.940 INFO:teuthology.orchestra.run.smithi070.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:06:16.940 INFO:teuthology.orchestra.run.smithi070.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T19:06:16.941 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:06:16.941 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-07-14T19:06:17.074 DEBUG:teuthology.parallel:result is None 2023-07-14T19:06:20.334 INFO:teuthology.orchestra.run.smithi104.stdout:Installed products updated. 2023-07-14T19:06:20.385 INFO:teuthology.orchestra.run.smithi053.stdout:Installed products updated. 2023-07-14T19:06:20.525 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:06:20.525 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2023-07-14T19:06:20.525 INFO:teuthology.orchestra.run.smithi104.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:20.525 INFO:teuthology.orchestra.run.smithi104.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:06:20.526 INFO:teuthology.orchestra.run.smithi104.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:20.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T19:06:20.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T19:06:20.526 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T19:06:20.526 INFO:teuthology.orchestra.run.smithi104.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:20.526 INFO:teuthology.orchestra.run.smithi104.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:20.526 INFO:teuthology.orchestra.run.smithi104.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:06:20.527 INFO:teuthology.orchestra.run.smithi104.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T19:06:20.527 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:06:20.527 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2023-07-14T19:06:20.596 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:06:20.596 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-07-14T19:06:20.597 INFO:teuthology.orchestra.run.smithi053.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:20.597 INFO:teuthology.orchestra.run.smithi053.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:06:20.597 INFO:teuthology.orchestra.run.smithi053.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:20.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T19:06:20.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T19:06:20.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T19:06:20.598 INFO:teuthology.orchestra.run.smithi053.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:20.598 INFO:teuthology.orchestra.run.smithi053.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:06:20.598 INFO:teuthology.orchestra.run.smithi053.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:06:20.598 INFO:teuthology.orchestra.run.smithi053.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T19:06:20.598 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:06:20.598 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-07-14T19:06:20.653 DEBUG:teuthology.parallel:result is None 2023-07-14T19:06:20.734 DEBUG:teuthology.parallel:result is None 2023-07-14T19:06:20.734 INFO:teuthology.run_tasks:Running task install... 2023-07-14T19:06:20.744 DEBUG:teuthology.task.install:project ceph 2023-07-14T19:06:20.744 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'}} 2023-07-14T19:06:20.744 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'} 2023-07-14T19:06:20.745 INFO:teuthology.task.install:Using flavor: default 2023-07-14T19:06:20.751 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-14T19:06:20.751 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-14T19:06:20.752 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T19:06:20.752 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:06:20.755 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T19:06:20.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:06:20.756 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T19:06:20.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:06:21.166 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T19:06:21.167 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T19:06:26.176 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T19:06:26.177 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T19:06:26.178 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T19:06:26.179 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T19:06:31.556 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T19:06:31.557 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:06:31.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T19:06:31.592 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T19:06:31.593 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T19:06:31.634 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T19:06:31.635 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:06:31.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T19:06:31.649 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T19:06:31.649 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:06:31.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T19:06:31.667 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T19:06:31.669 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T19:06:31.669 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T19:06:31.683 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T19:06:31.683 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T19:06:31.739 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T19:06:31.751 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T19:06:31.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T19:06:31.821 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T19:06:31.823 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2023-07-14T19:06:31.824 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2023-07-14T19:06:31.834 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T19:06:31.889 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2023-07-14T19:06:31.890 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2023-07-14T19:06:31.901 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2023-07-14T19:06:31.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2023-07-14T19:06:32.225 INFO:teuthology.orchestra.run.smithi053.stdout:Updating Subscription Management repositories. 2023-07-14T19:06:32.262 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2023-07-14T19:06:32.276 INFO:teuthology.orchestra.run.smithi070.stdout:Updating Subscription Management repositories. 2023-07-14T19:06:33.138 INFO:teuthology.orchestra.run.smithi104.stdout:44 files removed 2023-07-14T19:06:33.189 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T19:06:33.214 INFO:teuthology.orchestra.run.smithi053.stdout:44 files removed 2023-07-14T19:06:33.234 INFO:teuthology.orchestra.run.smithi070.stdout:44 files removed 2023-07-14T19:06:33.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T19:06:33.284 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T19:06:33.521 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2023-07-14T19:06:33.613 INFO:teuthology.orchestra.run.smithi070.stdout:Updating Subscription Management repositories. 2023-07-14T19:06:33.622 INFO:teuthology.orchestra.run.smithi053.stdout:Updating Subscription Management repositories. 2023-07-14T19:06:35.490 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 95 kB/s | 81 kB 00:00 2023-07-14T19:06:35.635 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 103 kB/s | 81 kB 00:00 2023-07-14T19:06:36.232 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-07-14T19:06:36.394 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-07-14T19:06:36.879 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 40 kB/s | 81 kB 00:02 2023-07-14T19:06:37.199 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 1.6 kB/s | 1.5 kB 00:00 2023-07-14T19:06:37.220 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 1.9 kB/s | 1.5 kB 00:00 2023-07-14T19:06:37.691 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for python3-asyncssh owned by ceph 7.3 kB/s | 3.5 kB 00:00 2023-07-14T19:06:37.720 INFO:teuthology.orchestra.run.smithi070.stdout:Copr repo for python3-asyncssh owned by ceph 7.2 kB/s | 3.5 kB 00:00 2023-07-14T19:06:37.746 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 11 kB/s | 9.4 kB 00:00 2023-07-14T19:06:39.081 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 1.1 kB/s | 1.5 kB 00:01 2023-07-14T19:06:39.631 INFO:teuthology.orchestra.run.smithi053.stdout:Copr repo for python3-asyncssh owned by ceph 6.6 kB/s | 3.5 kB 00:00 2023-07-14T19:06:40.650 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 16 MB 00:02 2023-07-14T19:06:40.829 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 5.1 MB/s | 16 MB 00:03 2023-07-14T19:06:44.304 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 233 kB/s | 24 kB 00:00 2023-07-14T19:06:44.325 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 149 kB/s | 24 kB 00:00 2023-07-14T19:06:45.386 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 MB/s | 47 MB 00:00 2023-07-14T19:06:45.623 INFO:teuthology.orchestra.run.smithi070.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 MB/s | 53 MB 00:01 2023-07-14T19:06:53.619 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 53 MB 00:01 2023-07-14T19:06:53.889 INFO:teuthology.orchestra.run.smithi070.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 MB/s | 47 MB 00:01 2023-07-14T19:07:04.778 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:07:04.779 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:07:04.793 INFO:teuthology.orchestra.run.smithi070.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T19:07:04.794 INFO:teuthology.orchestra.run.smithi070.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T19:07:04.795 INFO:teuthology.orchestra.run.smithi070.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T19:07:04.796 INFO:teuthology.orchestra.run.smithi070.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T19:07:04.798 INFO:teuthology.orchestra.run.smithi070.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T19:07:04.801 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T19:07:04.802 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T19:07:04.810 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:07:04.811 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:07:04.825 INFO:teuthology.orchestra.run.smithi104.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T19:07:04.826 INFO:teuthology.orchestra.run.smithi104.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T19:07:04.827 INFO:teuthology.orchestra.run.smithi104.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T19:07:04.828 INFO:teuthology.orchestra.run.smithi104.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T19:07:04.831 INFO:teuthology.orchestra.run.smithi104.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T19:07:04.834 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T19:07:04.835 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T19:07:04.935 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2023-07-14T19:07:04.947 INFO:teuthology.orchestra.run.smithi070.stdout:=========================================================================================================================== 2023-07-14T19:07:04.948 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2023-07-14T19:07:04.948 INFO:teuthology.orchestra.run.smithi070.stdout:=========================================================================================================================== 2023-07-14T19:07:04.948 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2023-07-14T19:07:04.948 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T19:07:04.948 INFO:teuthology.orchestra.run.smithi070.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T19:07:04.949 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T19:07:04.949 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T19:07:04.949 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T19:07:04.949 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T19:07:04.949 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T19:07:04.949 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T19:07:04.949 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T19:07:04.950 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T19:07:04.950 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T19:07:04.950 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T19:07:04.950 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T19:07:04.950 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T19:07:04.950 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T19:07:04.950 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T19:07:04.950 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T19:07:04.951 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T19:07:04.951 INFO:teuthology.orchestra.run.smithi070.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T19:07:04.951 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T19:07:04.951 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T19:07:04.951 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T19:07:04.951 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T19:07:04.951 INFO:teuthology.orchestra.run.smithi070.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T19:07:04.952 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T19:07:04.952 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T19:07:04.952 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T19:07:04.952 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T19:07:04.952 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T19:07:04.952 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T19:07:04.952 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T19:07:04.953 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T19:07:04.953 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T19:07:04.953 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T19:07:04.953 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T19:07:04.953 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T19:07:04.953 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T19:07:04.953 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2023-07-14T19:07:04.953 INFO:teuthology.orchestra.run.smithi070.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T19:07:04.954 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T19:07:04.954 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T19:07:04.954 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2023-07-14T19:07:04.954 INFO:teuthology.orchestra.run.smithi070.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T19:07:04.954 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T19:07:04.954 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T19:07:04.954 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T19:07:04.955 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T19:07:04.955 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T19:07:04.955 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T19:07:04.955 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T19:07:04.955 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T19:07:04.955 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T19:07:04.955 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T19:07:04.956 INFO:teuthology.orchestra.run.smithi070.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T19:07:04.956 INFO:teuthology.orchestra.run.smithi070.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T19:07:04.956 INFO:teuthology.orchestra.run.smithi070.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T19:07:04.956 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T19:07:04.956 INFO:teuthology.orchestra.run.smithi070.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T19:07:04.956 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T19:07:04.956 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T19:07:04.956 INFO:teuthology.orchestra.run.smithi070.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T19:07:04.957 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T19:07:04.957 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T19:07:04.957 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T19:07:04.957 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T19:07:04.957 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T19:07:04.957 INFO:teuthology.orchestra.run.smithi070.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T19:07:04.957 INFO:teuthology.orchestra.run.smithi070.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T19:07:04.958 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T19:07:04.958 INFO:teuthology.orchestra.run.smithi070.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T19:07:04.958 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T19:07:04.958 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T19:07:04.958 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T19:07:04.958 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T19:07:04.958 INFO:teuthology.orchestra.run.smithi070.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T19:07:04.958 INFO:teuthology.orchestra.run.smithi070.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T19:07:04.959 INFO:teuthology.orchestra.run.smithi070.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T19:07:04.959 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T19:07:04.959 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T19:07:04.959 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T19:07:04.959 INFO:teuthology.orchestra.run.smithi070.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T19:07:04.959 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T19:07:04.959 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T19:07:04.959 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T19:07:04.960 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T19:07:04.960 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T19:07:04.960 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T19:07:04.960 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T19:07:04.960 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T19:07:04.960 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T19:07:04.960 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T19:07:04.961 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T19:07:04.961 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T19:07:04.961 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T19:07:04.961 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T19:07:04.961 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:07:04.961 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T19:07:04.961 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T19:07:04.962 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T19:07:04.962 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T19:07:04.962 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T19:07:04.962 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T19:07:04.962 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T19:07:04.962 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T19:07:04.962 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T19:07:04.962 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T19:07:04.963 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T19:07:04.963 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T19:07:04.963 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T19:07:04.963 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T19:07:04.963 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T19:07:04.963 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T19:07:04.964 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T19:07:04.964 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T19:07:04.964 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:07:04.964 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T19:07:04.964 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T19:07:04.964 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T19:07:04.964 INFO:teuthology.orchestra.run.smithi070.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T19:07:04.965 INFO:teuthology.orchestra.run.smithi070.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T19:07:04.965 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T19:07:04.965 INFO:teuthology.orchestra.run.smithi070.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T19:07:04.965 INFO:teuthology.orchestra.run.smithi070.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T19:07:04.965 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T19:07:04.965 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T19:07:04.966 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T19:07:04.966 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T19:07:04.966 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T19:07:04.966 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T19:07:04.966 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T19:07:04.966 INFO:teuthology.orchestra.run.smithi070.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T19:07:04.966 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T19:07:04.967 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T19:07:04.967 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling module streams: 2023-07-14T19:07:04.967 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql 10 2023-07-14T19:07:04.967 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:07:04.967 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2023-07-14T19:07:04.968 INFO:teuthology.orchestra.run.smithi070.stdout:=========================================================================================================================== 2023-07-14T19:07:04.968 INFO:teuthology.orchestra.run.smithi070.stdout:Install 128 Packages 2023-07-14T19:07:04.968 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 3 Packages 2023-07-14T19:07:04.968 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:07:04.969 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2023-07-14T19:07:04.969 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 206 M 2023-07-14T19:07:04.970 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2023-07-14T19:07:04.977 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================================================== 2023-07-14T19:07:04.977 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2023-07-14T19:07:04.977 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================================================== 2023-07-14T19:07:04.977 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2023-07-14T19:07:04.977 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T19:07:04.977 INFO:teuthology.orchestra.run.smithi104.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T19:07:04.978 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T19:07:04.978 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T19:07:04.978 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T19:07:04.978 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T19:07:04.978 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T19:07:04.978 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T19:07:04.978 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T19:07:04.979 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T19:07:04.979 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T19:07:04.979 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T19:07:04.979 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T19:07:04.979 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T19:07:04.980 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T19:07:04.980 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T19:07:04.980 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T19:07:04.980 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T19:07:04.980 INFO:teuthology.orchestra.run.smithi104.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T19:07:04.980 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T19:07:04.980 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T19:07:04.981 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T19:07:04.981 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T19:07:04.981 INFO:teuthology.orchestra.run.smithi104.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T19:07:04.981 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T19:07:04.981 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T19:07:04.981 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T19:07:04.982 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T19:07:04.982 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T19:07:04.982 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T19:07:04.982 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T19:07:04.982 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T19:07:04.982 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T19:07:04.982 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T19:07:04.983 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T19:07:04.983 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T19:07:04.983 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T19:07:04.983 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2023-07-14T19:07:04.983 INFO:teuthology.orchestra.run.smithi104.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T19:07:04.983 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T19:07:04.983 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T19:07:04.983 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2023-07-14T19:07:04.984 INFO:teuthology.orchestra.run.smithi104.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T19:07:04.984 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T19:07:04.984 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T19:07:04.984 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T19:07:04.984 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T19:07:04.984 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T19:07:04.984 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T19:07:04.985 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T19:07:04.985 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T19:07:04.985 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T19:07:04.985 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T19:07:04.985 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T19:07:04.985 INFO:teuthology.orchestra.run.smithi104.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T19:07:04.985 INFO:teuthology.orchestra.run.smithi104.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T19:07:04.986 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T19:07:04.986 INFO:teuthology.orchestra.run.smithi104.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T19:07:04.986 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T19:07:04.986 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T19:07:04.986 INFO:teuthology.orchestra.run.smithi104.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T19:07:04.986 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T19:07:04.986 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T19:07:04.986 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T19:07:04.987 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T19:07:04.987 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T19:07:04.987 INFO:teuthology.orchestra.run.smithi104.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T19:07:04.987 INFO:teuthology.orchestra.run.smithi104.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T19:07:04.987 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T19:07:04.987 INFO:teuthology.orchestra.run.smithi104.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T19:07:04.987 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T19:07:04.988 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T19:07:04.988 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T19:07:04.988 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T19:07:04.988 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T19:07:04.988 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T19:07:04.988 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T19:07:04.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T19:07:04.988 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T19:07:04.989 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T19:07:04.989 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T19:07:04.989 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T19:07:04.989 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T19:07:04.989 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T19:07:04.989 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T19:07:04.989 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T19:07:04.990 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T19:07:04.990 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T19:07:04.990 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T19:07:04.990 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T19:07:04.990 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T19:07:04.990 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T19:07:04.991 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T19:07:04.991 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T19:07:04.991 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T19:07:04.991 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:07:04.991 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T19:07:04.991 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T19:07:04.992 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T19:07:04.992 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T19:07:04.992 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T19:07:04.992 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T19:07:04.992 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T19:07:04.992 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T19:07:04.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T19:07:04.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T19:07:04.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T19:07:04.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T19:07:04.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T19:07:04.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T19:07:04.993 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T19:07:04.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T19:07:04.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T19:07:04.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T19:07:04.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:07:04.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T19:07:04.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T19:07:04.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T19:07:04.994 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T19:07:04.995 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T19:07:04.995 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T19:07:04.995 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T19:07:04.995 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T19:07:04.995 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T19:07:04.995 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T19:07:04.995 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T19:07:04.996 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T19:07:04.996 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T19:07:04.996 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T19:07:04.996 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T19:07:04.996 INFO:teuthology.orchestra.run.smithi104.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T19:07:04.996 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T19:07:04.996 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T19:07:04.996 INFO:teuthology.orchestra.run.smithi104.stdout:Enabling module streams: 2023-07-14T19:07:04.997 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql 10 2023-07-14T19:07:04.997 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:07:04.997 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2023-07-14T19:07:04.997 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================================================== 2023-07-14T19:07:04.997 INFO:teuthology.orchestra.run.smithi104.stdout:Install 128 Packages 2023-07-14T19:07:04.997 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 3 Packages 2023-07-14T19:07:04.997 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:07:04.998 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 206 M 2023-07-14T19:07:04.998 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2023-07-14T19:07:05.835 INFO:teuthology.orchestra.run.smithi104.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 13 kB/s | 6.4 kB 00:00 2023-07-14T19:07:06.172 INFO:teuthology.orchestra.run.smithi070.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 12 kB/s | 6.4 kB 00:00 2023-07-14T19:07:07.460 INFO:teuthology.orchestra.run.smithi104.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 523 kB/s | 850 kB 00:01 2023-07-14T19:07:07.497 INFO:teuthology.orchestra.run.smithi070.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 642 kB/s | 850 kB 00:01 2023-07-14T19:07:07.727 INFO:teuthology.orchestra.run.smithi104.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 626 kB/s | 167 kB 00:00 2023-07-14T19:07:07.755 INFO:teuthology.orchestra.run.smithi070.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 647 kB/s | 167 kB 00:00 2023-07-14T19:07:09.006 INFO:teuthology.orchestra.run.smithi070.stdout:(4/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 1.6 MB/s | 5.4 MB 00:03 2023-07-14T19:07:09.997 INFO:teuthology.orchestra.run.smithi070.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 1.6 MB/s | 1.6 MB 00:00 2023-07-14T19:07:10.172 INFO:teuthology.orchestra.run.smithi070.stdout:(6/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 973 kB/s | 2.3 MB 00:02 2023-07-14T19:07:10.185 INFO:teuthology.orchestra.run.smithi104.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 956 kB/s | 2.3 MB 00:02 2023-07-14T19:07:12.046 INFO:teuthology.orchestra.run.smithi104.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 853 kB/s | 1.6 MB 00:01 2023-07-14T19:07:12.814 INFO:teuthology.orchestra.run.smithi070.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 1.6 MB/s | 4.6 MB 00:02 2023-07-14T19:07:13.960 INFO:teuthology.orchestra.run.smithi104.stdout:(6/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 639 kB/s | 5.4 MB 00:08 2023-07-14T19:07:17.285 INFO:teuthology.orchestra.run.smithi104.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 903 kB/s | 4.6 MB 00:05 2023-07-14T19:07:17.472 INFO:teuthology.orchestra.run.smithi070.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 1.9 MB/s | 22 MB 00:11 2023-07-14T19:07:17.597 INFO:teuthology.orchestra.run.smithi070.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 195 kB/s | 24 kB 00:00 2023-07-14T19:07:19.918 INFO:teuthology.orchestra.run.smithi104.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 1.5 MB/s | 22 MB 00:14 2023-07-14T19:07:20.068 INFO:teuthology.orchestra.run.smithi104.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 162 kB/s | 24 kB 00:00 2023-07-14T19:07:22.863 INFO:teuthology.orchestra.run.smithi070.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 1.3 MB/s | 14 MB 00:10 2023-07-14T19:07:23.688 INFO:teuthology.orchestra.run.smithi070.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 315 kB/s | 260 kB 00:00 2023-07-14T19:07:23.797 INFO:teuthology.orchestra.run.smithi070.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 286 kB/s | 31 kB 00:00 2023-07-14T19:07:24.555 INFO:teuthology.orchestra.run.smithi070.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 1.0 MB/s | 772 kB 00:00 2023-07-14T19:07:24.822 INFO:teuthology.orchestra.run.smithi070.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 508 kB/s | 135 kB 00:00 2023-07-14T19:07:25.114 INFO:teuthology.orchestra.run.smithi070.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 477 kB/s | 139 kB 00:00 2023-07-14T19:07:25.648 INFO:teuthology.orchestra.run.smithi070.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 887 kB/s | 473 kB 00:00 2023-07-14T19:07:29.585 INFO:teuthology.orchestra.run.smithi104.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 1.1 MB/s | 14 MB 00:12 2023-07-14T19:07:29.927 INFO:teuthology.orchestra.run.smithi104.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 762 kB/s | 260 kB 00:00 2023-07-14T19:07:30.036 INFO:teuthology.orchestra.run.smithi104.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 286 kB/s | 31 kB 00:00 2023-07-14T19:07:30.508 INFO:teuthology.orchestra.run.smithi070.stdout:(17/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 1.4 MB/s | 6.8 MB 00:04 2023-07-14T19:07:30.613 INFO:teuthology.orchestra.run.smithi070.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 425 kB/s | 45 kB 00:00 2023-07-14T19:07:30.711 INFO:teuthology.orchestra.run.smithi104.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 1.1 MB/s | 772 kB 00:00 2023-07-14T19:07:30.797 INFO:teuthology.orchestra.run.smithi070.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 668 kB/s | 122 kB 00:00 2023-07-14T19:07:30.878 INFO:teuthology.orchestra.run.smithi104.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 813 kB/s | 135 kB 00:00 2023-07-14T19:07:31.020 INFO:teuthology.orchestra.run.smithi104.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 983 kB/s | 139 kB 00:00 2023-07-14T19:07:31.147 INFO:teuthology.orchestra.run.smithi070.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 563 kB/s | 197 kB 00:00 2023-07-14T19:07:31.520 INFO:teuthology.orchestra.run.smithi104.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 946 kB/s | 473 kB 00:00 2023-07-14T19:07:31.602 INFO:teuthology.orchestra.run.smithi070.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 817 kB/s | 371 kB 00:00 2023-07-14T19:07:31.906 INFO:teuthology.orchestra.run.smithi070.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 1.2 MB/s | 364 kB 00:00 2023-07-14T19:07:32.015 INFO:teuthology.orchestra.run.smithi070.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 985 kB/s | 107 kB 00:00 2023-07-14T19:07:32.133 INFO:teuthology.orchestra.run.smithi070.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 803 kB/s | 94 kB 00:00 2023-07-14T19:07:32.815 INFO:teuthology.orchestra.run.smithi070.stdout:(25/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 803 kB/s | 18 MB 00:22 2023-07-14T19:07:33.016 INFO:teuthology.orchestra.run.smithi070.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 896 kB/s | 179 kB 00:00 2023-07-14T19:07:33.124 INFO:teuthology.orchestra.run.smithi070.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 213 kB/s | 23 kB 00:00 2023-07-14T19:07:33.128 INFO:teuthology.orchestra.run.smithi104.stdout:(17/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 949 kB/s | 18 MB 00:19 2023-07-14T19:07:33.220 INFO:teuthology.orchestra.run.smithi104.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 490 kB/s | 45 kB 00:00 2023-07-14T19:07:33.299 INFO:teuthology.orchestra.run.smithi070.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 739 kB/s | 129 kB 00:00 2023-07-14T19:07:33.387 INFO:teuthology.orchestra.run.smithi104.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 735 kB/s | 122 kB 00:00 2023-07-14T19:07:33.579 INFO:teuthology.orchestra.run.smithi104.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 1.0 MB/s | 197 kB 00:00 2023-07-14T19:07:34.071 INFO:teuthology.orchestra.run.smithi104.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 755 kB/s | 371 kB 00:00 2023-07-14T19:07:34.454 INFO:teuthology.orchestra.run.smithi104.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 951 kB/s | 364 kB 00:00 2023-07-14T19:07:34.596 INFO:teuthology.orchestra.run.smithi104.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 753 kB/s | 107 kB 00:00 2023-07-14T19:07:34.674 INFO:teuthology.orchestra.run.smithi070.stdout:(29/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.2 MB/s | 1.7 MB 00:01 2023-07-14T19:07:34.721 INFO:teuthology.orchestra.run.smithi104.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 750 kB/s | 94 kB 00:00 2023-07-14T19:07:34.825 INFO:teuthology.orchestra.run.smithi070.stdout:(30/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 1.3 MB/s | 3.5 MB 00:02 2023-07-14T19:07:35.033 INFO:teuthology.orchestra.run.smithi070.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 1.1 MB/s | 241 kB 00:00 2023-07-14T19:07:35.158 INFO:teuthology.orchestra.run.smithi070.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 378 kB/s | 47 kB 00:00 2023-07-14T19:07:35.292 INFO:teuthology.orchestra.run.smithi070.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 109 kB/s | 15 kB 00:00 2023-07-14T19:07:35.625 INFO:teuthology.orchestra.run.smithi070.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 775 kB/s | 258 kB 00:00 2023-07-14T19:07:35.860 INFO:teuthology.orchestra.run.smithi070.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 709 kB/s | 165 kB 00:00 2023-07-14T19:07:35.976 INFO:teuthology.orchestra.run.smithi070.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 154 kB/s | 18 kB 00:00 2023-07-14T19:07:36.768 INFO:teuthology.orchestra.run.smithi070.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 387 kB/s | 306 kB 00:00 2023-07-14T19:07:36.879 INFO:teuthology.orchestra.run.smithi104.stdout:(25/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 1.3 MB/s | 6.8 MB 00:05 2023-07-14T19:07:36.894 INFO:teuthology.orchestra.run.smithi070.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 470 kB/s | 59 kB 00:00 2023-07-14T19:07:37.038 INFO:teuthology.orchestra.run.smithi070.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 520 kB/s | 75 kB 00:00 2023-07-14T19:07:37.096 INFO:teuthology.orchestra.run.smithi104.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 827 kB/s | 179 kB 00:00 2023-07-14T19:07:37.221 INFO:teuthology.orchestra.run.smithi104.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 184 kB/s | 23 kB 00:00 2023-07-14T19:07:37.355 INFO:teuthology.orchestra.run.smithi104.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 971 kB/s | 129 kB 00:00 2023-07-14T19:07:37.555 INFO:teuthology.orchestra.run.smithi104.stdout:(29/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 1.2 MB/s | 3.5 MB 00:02 2023-07-14T19:07:37.719 INFO:teuthology.orchestra.run.smithi070.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 699 kB/s | 476 kB 00:00 2023-07-14T19:07:37.844 INFO:teuthology.orchestra.run.smithi070.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 350 kB/s | 44 kB 00:00 2023-07-14T19:07:38.144 INFO:teuthology.orchestra.run.smithi070.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 617 kB/s | 185 kB 00:00 2023-07-14T19:07:38.311 INFO:teuthology.orchestra.run.smithi070.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 198 kB/s | 33 kB 00:00 2023-07-14T19:07:38.462 INFO:teuthology.orchestra.run.smithi070.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 109 kB/s | 16 kB 00:00 2023-07-14T19:07:38.720 INFO:teuthology.orchestra.run.smithi070.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 668 kB/s | 173 kB 00:00 2023-07-14T19:07:38.814 INFO:teuthology.orchestra.run.smithi104.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.2 MB/s | 1.7 MB 00:01 2023-07-14T19:07:39.081 INFO:teuthology.orchestra.run.smithi104.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 903 kB/s | 241 kB 00:00 2023-07-14T19:07:39.231 INFO:teuthology.orchestra.run.smithi104.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 316 kB/s | 47 kB 00:00 2023-07-14T19:07:39.271 INFO:teuthology.orchestra.run.smithi070.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 699 kB/s | 384 kB 00:00 2023-07-14T19:07:39.390 INFO:teuthology.orchestra.run.smithi104.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 92 kB/s | 15 kB 00:00 2023-07-14T19:07:39.446 INFO:teuthology.orchestra.run.smithi070.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 465 kB/s | 81 kB 00:00 2023-07-14T19:07:39.613 INFO:teuthology.orchestra.run.smithi070.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 67 kB/s | 11 kB 00:00 2023-07-14T19:07:39.714 INFO:teuthology.orchestra.run.smithi104.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 795 kB/s | 258 kB 00:00 2023-07-14T19:07:39.788 INFO:teuthology.orchestra.run.smithi070.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 100 kB/s | 18 kB 00:00 2023-07-14T19:07:39.982 INFO:teuthology.orchestra.run.smithi104.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 620 kB/s | 165 kB 00:00 2023-07-14T19:07:40.098 INFO:teuthology.orchestra.run.smithi104.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 154 kB/s | 18 kB 00:00 2023-07-14T19:07:40.346 INFO:teuthology.orchestra.run.smithi070.stdout:(50/131): ceph-mgr-diskprediction-local-18.0.0- 1.3 MB/s | 7.4 MB 00:05 2023-07-14T19:07:40.688 INFO:teuthology.orchestra.run.smithi070.stdout:(51/131): python3-kubernetes-11.0.0-6.el8.noarc 1.3 MB/s | 1.2 MB 00:00 2023-07-14T19:07:40.699 INFO:teuthology.orchestra.run.smithi104.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 510 kB/s | 306 kB 00:00 2023-07-14T19:07:40.797 INFO:teuthology.orchestra.run.smithi070.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 108 kB/s | 49 kB 00:00 2023-07-14T19:07:40.830 INFO:teuthology.orchestra.run.smithi070.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 414 kB/s | 59 kB 00:00 2023-07-14T19:07:40.857 INFO:teuthology.orchestra.run.smithi104.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 372 kB/s | 59 kB 00:00 2023-07-14T19:07:40.975 INFO:teuthology.orchestra.run.smithi104.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 642 kB/s | 75 kB 00:00 2023-07-14T19:07:41.039 INFO:teuthology.orchestra.run.smithi070.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 249 kB/s | 60 kB 00:00 2023-07-14T19:07:41.064 INFO:teuthology.orchestra.run.smithi070.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2023-07-14T19:07:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2023-07-14T19:07:41.207 INFO:teuthology.orchestra.run.smithi070.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 237 kB/s | 34 kB 00:00 2023-07-14T19:07:41.216 INFO:teuthology.orchestra.run.smithi104.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.9 MB/s | 476 kB 00:00 2023-07-14T19:07:41.349 INFO:teuthology.orchestra.run.smithi070.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 435 kB/s | 61 kB 00:00 2023-07-14T19:07:41.350 INFO:teuthology.orchestra.run.smithi104.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 327 kB/s | 44 kB 00:00 2023-07-14T19:07:41.484 INFO:teuthology.orchestra.run.smithi104.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2023-07-14T19:07:41.491 INFO:teuthology.orchestra.run.smithi070.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 132 kB/s | 19 kB 00:00 2023-07-14T19:07:41.541 INFO:teuthology.orchestra.run.smithi070.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 547 kB/s | 196 kB 00:00 2023-07-14T19:07:41.609 INFO:teuthology.orchestra.run.smithi104.stdout:(43/131): ceph-mgr-diskprediction-local-18.0.0- 1.8 MB/s | 7.4 MB 00:04 2023-07-14T19:07:41.634 INFO:teuthology.orchestra.run.smithi104.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 220 kB/s | 33 kB 00:00 2023-07-14T19:07:41.641 INFO:teuthology.orchestra.run.smithi070.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 159 kB/s | 24 kB 00:00 2023-07-14T19:07:41.691 INFO:teuthology.orchestra.run.smithi070.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 195 kB/s | 29 kB 00:00 2023-07-14T19:07:41.784 INFO:teuthology.orchestra.run.smithi104.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.1 MB/s | 173 kB 00:00 2023-07-14T19:07:41.800 INFO:teuthology.orchestra.run.smithi070.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 174 kB/s | 27 kB 00:00 2023-07-14T19:07:41.926 INFO:teuthology.orchestra.run.smithi104.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.7 MB/s | 384 kB 00:00 2023-07-14T19:07:41.968 INFO:teuthology.orchestra.run.smithi104.stdout:(47/131): python3-certifi-2018.10.15-7.el8.noar 45 kB/s | 16 kB 00:00 2023-07-14T19:07:42.043 INFO:teuthology.orchestra.run.smithi104.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 696 kB/s | 81 kB 00:00 2023-07-14T19:07:42.050 INFO:teuthology.orchestra.run.smithi070.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.0 MB/s | 251 kB 00:00 2023-07-14T19:07:42.094 INFO:teuthology.orchestra.run.smithi104.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 88 kB/s | 11 kB 00:00 2023-07-14T19:07:42.177 INFO:teuthology.orchestra.run.smithi104.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 131 kB/s | 18 kB 00:00 2023-07-14T19:07:42.200 INFO:teuthology.orchestra.run.smithi070.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 409 kB/s | 61 kB 00:00 2023-07-14T19:07:42.226 INFO:teuthology.orchestra.run.smithi070.stdout:(66/131): python3-waitress-1.4.3-1.el8.noarch.r 478 kB/s | 255 kB 00:00 2023-07-14T19:07:42.311 INFO:teuthology.orchestra.run.smithi104.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 366 kB/s | 49 kB 00:00 2023-07-14T19:07:42.351 INFO:teuthology.orchestra.run.smithi070.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 567 kB/s | 85 kB 00:00 2023-07-14T19:07:42.376 INFO:teuthology.orchestra.run.smithi070.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 153 kB/s | 23 kB 00:00 2023-07-14T19:07:42.402 INFO:teuthology.orchestra.run.smithi070.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 70 kB 00:00 2023-07-14T19:07:42.436 INFO:teuthology.orchestra.run.smithi104.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 469 kB/s | 59 kB 00:00 2023-07-14T19:07:42.595 INFO:teuthology.orchestra.run.smithi104.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 380 kB/s | 60 kB 00:00 2023-07-14T19:07:42.602 INFO:teuthology.orchestra.run.smithi070.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 234 kB/s | 47 kB 00:00 2023-07-14T19:07:42.729 INFO:teuthology.orchestra.run.smithi104.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2023-07-14T19:07:42.777 INFO:teuthology.orchestra.run.smithi070.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.1 MB/s | 200 kB 00:00 2023-07-14T19:07:42.829 INFO:teuthology.orchestra.run.smithi104.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2023-07-14T19:07:42.936 INFO:teuthology.orchestra.run.smithi070.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 269 kB/s | 43 kB 00:00 2023-07-14T19:07:42.937 INFO:teuthology.orchestra.run.smithi104.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 311 kB/s | 34 kB 00:00 2023-07-14T19:07:43.046 INFO:teuthology.orchestra.run.smithi104.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2023-07-14T19:07:43.111 INFO:teuthology.orchestra.run.smithi070.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2023-07-14T19:07:43.121 INFO:teuthology.orchestra.run.smithi104.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 820 kB/s | 61 kB 00:00 2023-07-14T19:07:43.221 INFO:teuthology.orchestra.run.smithi104.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 187 kB/s | 19 kB 00:00 2023-07-14T19:07:43.237 INFO:teuthology.orchestra.run.smithi070.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 385 kB/s | 48 kB 00:00 2023-07-14T19:07:43.313 INFO:teuthology.orchestra.run.smithi104.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 261 kB/s | 24 kB 00:00 2023-07-14T19:07:43.370 INFO:teuthology.orchestra.run.smithi070.stdout:(75/131): thrift-0.13.0-2.el8.x86_64.rpm 1.7 MB/s | 1.7 MB 00:01 2023-07-14T19:07:43.413 INFO:teuthology.orchestra.run.smithi104.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 292 kB/s | 29 kB 00:00 2023-07-14T19:07:43.530 INFO:teuthology.orchestra.run.smithi104.stdout:(62/131): python3-kubernetes-11.0.0-6.el8.noarc 831 kB/s | 1.2 MB 00:01 2023-07-14T19:07:43.547 INFO:teuthology.orchestra.run.smithi104.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 206 kB/s | 27 kB 00:00 2023-07-14T19:07:43.664 INFO:teuthology.orchestra.run.smithi104.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.1 MB/s | 251 kB 00:00 2023-07-14T19:07:43.764 INFO:teuthology.orchestra.run.smithi104.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 613 kB/s | 61 kB 00:00 2023-07-14T19:07:43.798 INFO:teuthology.orchestra.run.smithi104.stdout:(66/131): python3-waitress-1.4.3-1.el8.noarch.r 955 kB/s | 255 kB 00:00 2023-07-14T19:07:43.820 INFO:teuthology.orchestra.run.smithi070.stdout:(76/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 1.8 MB/s | 48 MB 00:26 2023-07-14T19:07:43.854 INFO:teuthology.orchestra.run.smithi070.stdout:(77/131): libsepol-devel-2.9-3.el8.x86_64.rpm 141 kB/s | 87 kB 00:00 2023-07-14T19:07:43.865 INFO:teuthology.orchestra.run.smithi104.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 852 kB/s | 85 kB 00:00 2023-07-14T19:07:43.916 INFO:teuthology.orchestra.run.smithi104.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 196 kB/s | 23 kB 00:00 2023-07-14T19:07:43.941 INFO:teuthology.orchestra.run.smithi104.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T19:07:44.012 INFO:teuthology.orchestra.run.smithi070.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 587 kB/s | 109 kB 00:00 2023-07-14T19:07:44.038 INFO:teuthology.orchestra.run.smithi070.stdout:(79/131): python3-requests-oauthlib-1.0.0-1.el8 64 kB/s | 43 kB 00:00 2023-07-14T19:07:44.063 INFO:teuthology.orchestra.run.smithi070.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.1 MB/s | 229 kB 00:00 2023-07-14T19:07:44.097 INFO:teuthology.orchestra.run.smithi070.stdout:(81/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 6.5 MB/s | 560 kB 00:00 2023-07-14T19:07:44.122 INFO:teuthology.orchestra.run.smithi070.stdout:(82/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 223 kB/s | 19 kB 00:00 2023-07-14T19:07:44.183 INFO:teuthology.orchestra.run.smithi104.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 5.5 MB/s | 1.7 MB 00:00 2023-07-14T19:07:44.197 INFO:teuthology.orchestra.run.smithi070.stdout:(83/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.1 MB/s | 155 kB 00:00 2023-07-14T19:07:44.239 INFO:teuthology.orchestra.run.smithi070.stdout:(84/131): python3-cryptography-3.2.1-5.el8.x86_ 3.8 MB/s | 559 kB 00:00 2023-07-14T19:07:44.316 INFO:teuthology.orchestra.run.smithi104.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 827 kB/s | 110 kB 00:00 2023-07-14T19:07:44.322 INFO:teuthology.orchestra.run.smithi070.stdout:(85/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 89 kB/s | 18 kB 00:00 2023-07-14T19:07:44.381 INFO:teuthology.orchestra.run.smithi070.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.6 MB/s | 238 kB 00:00 2023-07-14T19:07:44.431 INFO:teuthology.orchestra.run.smithi070.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 2.5 MB/s | 605 kB 00:00 2023-07-14T19:07:44.475 INFO:teuthology.orchestra.run.smithi104.stdout:(72/131): python3-scipy-1.0.0-21.module+el8.5.0 26 MB/s | 14 MB 00:00 2023-07-14T19:07:44.500 INFO:teuthology.orchestra.run.smithi104.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 295 kB/s | 54 kB 00:00 2023-07-14T19:07:44.507 INFO:teuthology.orchestra.run.smithi070.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 13 MB/s | 2.3 MB 00:00 2023-07-14T19:07:44.557 INFO:teuthology.orchestra.run.smithi070.stdout:(89/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 412 kB/s | 72 kB 00:00 2023-07-14T19:07:44.607 INFO:teuthology.orchestra.run.smithi070.stdout:(90/131): fuse-2.9.7-15.el8.x86_64.rpm 471 kB/s | 83 kB 00:00 2023-07-14T19:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 358 kB/s | 60 kB 00:00 2023-07-14T19:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 690 kB/s | 98 kB 00:00 2023-07-14T19:07:44.767 INFO:teuthology.orchestra.run.smithi070.stdout:(93/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 363 kB/s | 58 kB 00:00 2023-07-14T19:07:44.808 INFO:teuthology.orchestra.run.smithi070.stdout:(94/131): libcom_err-devel-1.45.6-4.el8.x86_64. 288 kB/s | 39 kB 00:00 2023-07-14T19:07:44.867 INFO:teuthology.orchestra.run.smithi070.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.8 MB/s | 644 kB 00:00 2023-07-14T19:07:44.934 INFO:teuthology.orchestra.run.smithi104.stdout:(74/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 1.9 MB/s | 48 MB 00:24 2023-07-14T19:07:44.943 INFO:teuthology.orchestra.run.smithi070.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 974 kB/s | 171 kB 00:00 2023-07-14T19:07:44.967 INFO:teuthology.orchestra.run.smithi104.stdout:(75/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 526 kB/s | 259 kB 00:00 2023-07-14T19:07:44.976 INFO:teuthology.orchestra.run.smithi070.stdout:(97/131): NetworkManager-initscripts-updown-1.3 823 kB/s | 138 kB 00:00 2023-07-14T19:07:45.001 INFO:teuthology.orchestra.run.smithi104.stdout:(76/131): socat-1.7.4.1-1.el8.x86_64.rpm 645 kB/s | 323 kB 00:00 2023-07-14T19:07:45.093 INFO:teuthology.orchestra.run.smithi104.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 5.6 MB/s | 688 kB 00:00 2023-07-14T19:07:45.093 INFO:teuthology.orchestra.run.smithi070.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 733 kB/s | 110 kB 00:00 2023-07-14T19:07:45.118 INFO:teuthology.orchestra.run.smithi104.stdout:(78/131): python3-markupsafe-0.23-19.el8.x86_64 211 kB/s | 39 kB 00:00 2023-07-14T19:07:45.119 INFO:teuthology.orchestra.run.smithi070.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 382 kB/s | 54 kB 00:00 2023-07-14T19:07:45.143 INFO:teuthology.orchestra.run.smithi104.stdout:(79/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.7 MB/s | 538 kB 00:00 2023-07-14T19:07:45.185 INFO:teuthology.orchestra.run.smithi104.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2023-07-14T19:07:45.227 INFO:teuthology.orchestra.run.smithi104.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-07-14T19:07:45.278 INFO:teuthology.orchestra.run.smithi104.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 5.2 MB/s | 713 kB 00:00 2023-07-14T19:07:45.310 INFO:teuthology.orchestra.run.smithi070.stdout:(100/131): python3-scipy-1.0.0-21.module+el8.5. 31 MB/s | 14 MB 00:00 2023-07-14T19:07:45.335 INFO:teuthology.orchestra.run.smithi070.stdout:(101/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2023-07-14T19:07:45.360 INFO:teuthology.orchestra.run.smithi070.stdout:(102/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.3 MB/s | 323 kB 00:00 2023-07-14T19:07:45.361 INFO:teuthology.orchestra.run.smithi104.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 199 kB/s | 27 kB 00:00 2023-07-14T19:07:45.394 INFO:teuthology.orchestra.run.smithi104.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 2.7 MB/s | 320 kB 00:00 2023-07-14T19:07:45.461 INFO:teuthology.orchestra.run.smithi104.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2023-07-14T19:07:45.478 INFO:teuthology.orchestra.run.smithi070.stdout:(103/131): python3-markupsafe-0.23-19.el8.x86_6 232 kB/s | 39 kB 00:00 2023-07-14T19:07:45.511 INFO:teuthology.orchestra.run.smithi070.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.5 MB/s | 538 kB 00:00 2023-07-14T19:07:45.527 INFO:teuthology.orchestra.run.smithi104.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 265 kB/s | 44 kB 00:00 2023-07-14T19:07:45.545 INFO:teuthology.orchestra.run.smithi070.stdout:(105/131): bison-3.0.4-10.el8.x86_64.rpm 3.2 MB/s | 688 kB 00:00 2023-07-14T19:07:45.628 INFO:teuthology.orchestra.run.smithi104.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 16 MB/s | 3.7 MB 00:00 2023-07-14T19:07:45.645 INFO:teuthology.orchestra.run.smithi070.stdout:(106/131): python3-mako-1.0.6-13.el8.noarch.rpm 939 kB/s | 157 kB 00:00 2023-07-14T19:07:45.662 INFO:teuthology.orchestra.run.smithi104.stdout:(88/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.2 MB/s | 457 kB 00:00 2023-07-14T19:07:45.687 INFO:teuthology.orchestra.run.smithi070.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-07-14T19:07:45.688 INFO:teuthology.orchestra.run.smithi104.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.4 MB/s | 225 kB 00:00 2023-07-14T19:07:45.729 INFO:teuthology.orchestra.run.smithi070.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 3.8 MB/s | 713 kB 00:00 2023-07-14T19:07:45.754 INFO:teuthology.orchestra.run.smithi104.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 510 kB/s | 64 kB 00:00 2023-07-14T19:07:45.780 INFO:teuthology.orchestra.run.smithi104.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2023-07-14T19:07:45.804 INFO:teuthology.orchestra.run.smithi070.stdout:(109/131): python3-jsonpatch-1.21-2.el8.noarch. 227 kB/s | 27 kB 00:00 2023-07-14T19:07:45.830 INFO:teuthology.orchestra.run.smithi104.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 141 kB/s | 20 kB 00:00 2023-07-14T19:07:45.887 INFO:teuthology.orchestra.run.smithi070.stdout:(110/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-07-14T19:07:45.922 INFO:teuthology.orchestra.run.smithi104.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 619 kB/s | 103 kB 00:00 2023-07-14T19:07:45.938 INFO:teuthology.orchestra.run.smithi070.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.5 MB/s | 320 kB 00:00 2023-07-14T19:07:45.964 INFO:teuthology.orchestra.run.smithi104.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 3.8 MB/s | 709 kB 00:00 2023-07-14T19:07:45.988 INFO:teuthology.orchestra.run.smithi070.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 239 kB/s | 44 kB 00:00 2023-07-14T19:07:46.096 INFO:teuthology.orchestra.run.smithi104.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 18 MB/s | 4.9 MB 00:00 2023-07-14T19:07:46.097 INFO:teuthology.orchestra.run.smithi070.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 18 MB/s | 3.7 MB 00:00 2023-07-14T19:07:46.122 INFO:teuthology.orchestra.run.smithi070.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.6 MB/s | 225 kB 00:00 2023-07-14T19:07:46.131 INFO:teuthology.orchestra.run.smithi104.stdout:(96/131): libpq-13.5-1.el8.x86_64.rpm 948 kB/s | 198 kB 00:00 2023-07-14T19:07:46.147 INFO:teuthology.orchestra.run.smithi070.stdout:(115/131): python3-werkzeug-0.12.2-4.el8.noarch 2.1 MB/s | 457 kB 00:00 2023-07-14T19:07:46.206 INFO:teuthology.orchestra.run.smithi070.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 588 kB/s | 64 kB 00:00 2023-07-14T19:07:46.239 INFO:teuthology.orchestra.run.smithi104.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-07-14T19:07:46.265 INFO:teuthology.orchestra.run.smithi104.stdout:(98/131): jq-1.6-3.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-07-14T19:07:46.290 INFO:teuthology.orchestra.run.smithi104.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2023-07-14T19:07:46.348 INFO:teuthology.orchestra.run.smithi070.stdout:(117/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 560 kB/s | 126 kB 00:00 2023-07-14T19:07:46.398 INFO:teuthology.orchestra.run.smithi070.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 539 kB/s | 103 kB 00:00 2023-07-14T19:07:46.432 INFO:teuthology.orchestra.run.smithi070.stdout:(119/131): python3-jsonpointer-1.10-11.el8.noar 70 kB/s | 20 kB 00:00 2023-07-14T19:07:46.432 INFO:teuthology.orchestra.run.smithi104.stdout:(100/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-07-14T19:07:46.482 INFO:teuthology.orchestra.run.smithi104.stdout:(101/131): postgresql-contrib-10.21-2.module+el 3.6 MB/s | 808 kB 00:00 2023-07-14T19:07:46.583 INFO:teuthology.orchestra.run.smithi104.stdout:(102/131): postgresql-server-10.21-2.module+el8 15 MB/s | 5.1 MB 00:00 2023-07-14T19:07:46.607 INFO:teuthology.orchestra.run.smithi104.stdout:(103/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 267 kB/s | 47 kB 00:00 2023-07-14T19:07:46.633 INFO:teuthology.orchestra.run.smithi104.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.3 MB/s | 200 kB 00:00 2023-07-14T19:07:46.682 INFO:teuthology.orchestra.run.smithi070.stdout:(120/131): openblas-threads-0.3.15-3.el8.x86_64 17 MB/s | 4.9 MB 00:00 2023-07-14T19:07:46.700 INFO:teuthology.orchestra.run.smithi104.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 364 kB/s | 43 kB 00:00 2023-07-14T19:07:46.707 INFO:teuthology.orchestra.run.smithi070.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 718 kB/s | 198 kB 00:00 2023-07-14T19:07:46.725 INFO:teuthology.orchestra.run.smithi104.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.8 MB/s | 220 kB 00:00 2023-07-14T19:07:46.749 INFO:teuthology.orchestra.run.smithi070.stdout:(122/131): libtool-2.4.6-25.el8.x86_64.rpm 1.7 MB/s | 709 kB 00:00 2023-07-14T19:07:46.750 INFO:teuthology.orchestra.run.smithi104.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 410 kB/s | 48 kB 00:00 2023-07-14T19:07:46.801 INFO:teuthology.orchestra.run.smithi104.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 865 kB/s | 87 kB 00:00 2023-07-14T19:07:46.884 INFO:teuthology.orchestra.run.smithi104.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 269 kB/s | 43 kB 00:00 2023-07-14T19:07:46.935 INFO:teuthology.orchestra.run.smithi104.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.7 MB/s | 229 kB 00:00 2023-07-14T19:07:46.941 INFO:teuthology.orchestra.run.smithi070.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 866 kB/s | 202 kB 00:00 2023-07-14T19:07:46.985 INFO:teuthology.orchestra.run.smithi104.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 5.5 MB/s | 560 kB 00:00 2023-07-14T19:07:47.002 INFO:teuthology.orchestra.run.smithi104.stdout:(112/131): python3-pycparser-2.14-14.el8.noarch 435 kB/s | 109 kB 00:00 2023-07-14T19:07:47.060 INFO:teuthology.orchestra.run.smithi104.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 2.0 MB/s | 155 kB 00:00 2023-07-14T19:07:47.075 INFO:teuthology.orchestra.run.smithi070.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-07-14T19:07:47.094 INFO:teuthology.orchestra.run.smithi104.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 5.9 MB/s | 559 kB 00:00 2023-07-14T19:07:47.136 INFO:teuthology.orchestra.run.smithi104.stdout:(115/131): libaio-devel-0.3.112-1.el8.x86_64.rp 93 kB/s | 19 kB 00:00 2023-07-14T19:07:47.192 INFO:teuthology.orchestra.run.smithi070.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 800 kB/s | 354 kB 00:00 2023-07-14T19:07:47.251 INFO:teuthology.orchestra.run.smithi104.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 94 kB/s | 18 kB 00:00 2023-07-14T19:07:47.286 INFO:teuthology.orchestra.run.smithi104.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.1 MB/s | 605 kB 00:00 2023-07-14T19:07:47.325 INFO:teuthology.orchestra.run.smithi070.stdout:(126/131): postgresql-server-10.21-2.module+el8 13 MB/s | 5.1 MB 00:00 2023-07-14T19:07:47.354 INFO:teuthology.orchestra.run.smithi104.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2023-07-14T19:07:47.367 INFO:teuthology.orchestra.run.smithi070.stdout:(127/131): postgresql-contrib-10.21-2.module+el 2.7 MB/s | 808 kB 00:00 2023-07-14T19:07:47.418 INFO:teuthology.orchestra.run.smithi070.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 6.7 MB/s | 1.5 MB 00:00 2023-07-14T19:07:47.437 INFO:teuthology.orchestra.run.smithi104.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 13 MB/s | 2.3 MB 00:00 2023-07-14T19:07:47.462 INFO:teuthology.orchestra.run.smithi104.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 412 kB/s | 72 kB 00:00 2023-07-14T19:07:47.487 INFO:teuthology.orchestra.run.smithi104.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 619 kB/s | 83 kB 00:00 2023-07-14T19:07:47.554 INFO:teuthology.orchestra.run.smithi104.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 512 kB/s | 60 kB 00:00 2023-07-14T19:07:47.568 INFO:teuthology.orchestra.run.smithi070.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.2 MB/s | 798 kB 00:00 2023-07-14T19:07:47.638 INFO:teuthology.orchestra.run.smithi104.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 384 kB/s | 58 kB 00:00 2023-07-14T19:07:47.663 INFO:teuthology.orchestra.run.smithi104.stdout:(124/131): libuuid-devel-2.32.1-35.el8.x86_64.r 489 kB/s | 98 kB 00:00 2023-07-14T19:07:47.705 INFO:teuthology.orchestra.run.smithi104.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 257 kB/s | 39 kB 00:00 2023-07-14T19:07:47.738 INFO:teuthology.orchestra.run.smithi104.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.2 MB/s | 171 kB 00:00 2023-07-14T19:07:47.772 INFO:teuthology.orchestra.run.smithi104.stdout:(127/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.7 MB/s | 644 kB 00:00 2023-07-14T19:07:47.856 INFO:teuthology.orchestra.run.smithi104.stdout:(128/131): NetworkManager-initscripts-updown-1. 915 kB/s | 138 kB 00:00 2023-07-14T19:07:47.998 INFO:teuthology.orchestra.run.smithi104.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.5 MB/s | 798 kB 00:00 2023-07-14T19:07:48.559 INFO:teuthology.orchestra.run.smithi070.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 3.0 MB/s | 3.7 MB 00:01 2023-07-14T19:07:48.823 INFO:teuthology.orchestra.run.smithi104.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 3.5 MB/s | 3.6 MB 00:01 2023-07-14T19:07:49.431 INFO:teuthology.orchestra.run.smithi104.stdout:(131/131): librados2-18.0.0-4940.g81c9387d.el8. 2.2 MB/s | 3.7 MB 00:01 2023-07-14T19:07:49.434 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:07:49.434 INFO:teuthology.orchestra.run.smithi104.stdout:Total 4.6 MB/s | 206 MB 00:44 2023-07-14T19:07:49.626 INFO:teuthology.orchestra.run.smithi070.stdout:(131/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.6 MB/s | 3.6 MB 00:02 2023-07-14T19:07:49.629 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:07:49.629 INFO:teuthology.orchestra.run.smithi070.stdout:Total 4.6 MB/s | 206 MB 00:44 2023-07-14T19:07:50.255 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2023-07-14T19:07:50.374 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2023-07-14T19:07:50.375 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2023-07-14T19:07:50.456 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2023-07-14T19:07:50.575 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2023-07-14T19:07:50.576 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2023-07-14T19:07:51.483 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2023-07-14T19:07:51.487 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2023-07-14T19:07:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2023-07-14T19:07:51.772 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2023-07-14T19:07:56.311 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2023-07-14T19:07:56.520 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T19:07:56.523 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2023-07-14T19:07:56.694 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T19:07:56.695 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T19:07:56.903 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T19:07:56.911 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-14T19:07:57.064 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:07:57.113 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T19:07:57.172 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:07:57.201 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T19:07:57.397 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:07:57.430 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T19:07:57.781 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:07:57.833 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T19:07:58.006 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:07:58.013 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:07:58.402 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:07:58.407 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:07:58.581 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:07:58.588 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:07:58.707 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:07:58.713 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:07:58.889 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:07:58.896 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:07:58.994 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:07:59.033 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:07:59.172 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:07:59.176 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T19:07:59.433 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T19:07:59.583 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T19:07:59.624 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T19:07:59.791 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T19:08:00.003 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T19:08:00.169 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T19:08:00.266 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T19:08:00.400 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T19:08:00.421 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:08:00.538 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:08:00.557 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:08:00.639 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:08:00.696 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:08:00.788 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:08:00.793 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:08:00.929 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T19:08:00.936 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:08:01.047 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T19:08:01.183 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T19:08:01.229 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T19:08:01.378 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T19:08:01.561 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T19:08:01.696 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:08:01.827 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T19:08:01.831 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:08:01.996 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T19:08:01.996 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:08:02.578 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T19:08:02.609 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:08:02.771 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:08:02.795 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T19:08:03.350 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:08:03.501 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T19:08:03.537 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:08:03.760 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-14T19:08:03.979 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-14T19:08:04.190 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-14T19:08:04.216 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:08:04.368 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-14T19:08:04.477 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T19:08:04.609 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-14T19:08:04.676 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T19:08:04.847 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T19:08:04.914 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T19:08:05.010 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T19:08:05.077 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T19:08:05.164 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T19:08:05.208 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T19:08:05.441 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T19:08:05.653 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:08:06.063 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T19:08:06.410 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-14T19:08:06.663 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-14T19:08:06.958 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-14T19:08:07.155 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T19:08:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:08:08.537 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T19:08:08.843 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T19:08:08.905 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T19:08:09.097 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T19:08:09.254 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-14T19:08:09.331 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T19:08:09.464 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T19:08:09.555 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T19:08:09.637 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T19:08:09.746 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T19:08:09.849 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T19:08:09.950 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T19:08:10.006 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T19:08:10.102 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T19:08:10.151 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-14T19:08:10.271 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T19:08:10.371 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-14T19:08:10.420 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T19:08:10.555 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-14T19:08:10.594 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:08:10.653 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:08:10.698 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-14T19:08:10.897 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T19:08:10.944 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-14T19:08:11.131 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T19:08:11.151 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-14T19:08:11.286 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:08:11.355 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:08:11.518 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-14T19:08:11.564 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T19:08:11.731 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T19:08:11.788 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-14T19:08:11.901 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T19:08:11.978 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-14T19:08:12.070 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T19:08:12.230 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-14T19:08:12.238 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T19:08:12.363 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T19:08:12.473 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-14T19:08:12.583 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T19:08:12.780 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T19:08:12.812 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-14T19:08:13.011 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-14T19:08:13.136 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T19:08:13.227 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-14T19:08:13.412 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T19:08:13.431 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-14T19:08:13.597 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-14T19:08:13.638 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T19:08:13.788 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-14T19:08:13.846 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T19:08:14.056 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T19:08:14.084 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-14T19:08:14.368 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-14T19:08:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T19:08:14.593 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T19:08:14.622 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-14T19:08:14.777 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T19:08:14.951 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-14T19:08:15.005 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T19:08:15.172 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T19:08:15.317 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-14T19:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T19:08:15.478 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-14T19:08:15.642 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T19:08:15.691 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T19:08:15.717 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T19:08:15.955 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T19:08:15.956 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-14T19:08:16.187 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-14T19:08:16.190 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T19:08:16.326 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T19:08:16.402 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T19:08:16.461 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T19:08:16.605 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-14T19:08:16.634 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T19:08:16.785 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-14T19:08:16.812 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T19:08:16.917 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T19:08:17.088 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T19:08:17.088 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T19:08:17.316 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T19:08:17.374 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T19:08:17.497 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T19:08:17.584 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T19:08:17.653 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T19:08:17.722 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T19:08:17.842 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T19:08:17.896 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T19:08:18.028 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T19:08:18.095 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T19:08:18.219 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T19:08:18.289 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T19:08:18.387 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T19:08:18.496 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T19:08:18.531 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T19:08:18.653 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T19:08:18.775 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T19:08:18.798 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T19:08:19.013 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T19:08:19.062 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T19:08:19.215 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T19:08:19.271 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T19:08:19.439 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T19:08:19.466 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T19:08:19.700 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T19:08:19.707 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T19:08:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T19:08:19.908 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T19:08:20.058 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T19:08:20.093 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T19:08:20.285 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T19:08:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T19:08:21.395 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T19:08:21.428 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T19:08:21.662 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T19:08:21.683 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T19:08:21.916 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T19:08:22.025 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T19:08:22.218 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T19:08:22.275 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T19:08:22.470 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T19:08:22.546 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T19:08:22.746 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T19:08:23.257 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T19:08:23.416 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:08:23.471 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:08:23.521 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T19:08:23.625 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T19:08:23.691 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:08:23.779 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:08:23.933 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T19:08:27.248 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:08:27.457 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:08:27.578 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:08:27.782 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:08:28.101 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:08:28.361 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:08:28.384 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:08:28.474 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:08:28.607 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:08:28.638 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:08:28.698 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:08:28.865 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:08:47.270 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:08:47.271 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2023-07-14T19:08:47.271 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2023-07-14T19:08:47.271 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /dev 2023-07-14T19:08:47.271 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /run 2023-07-14T19:08:47.271 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2023-07-14T19:08:47.272 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2023-07-14T19:08:47.272 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2023-07-14T19:08:47.272 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2023-07-14T19:08:47.272 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /dev 2023-07-14T19:08:47.272 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:08:47.611 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:08:47.786 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:08:47.786 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2023-07-14T19:08:47.786 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2023-07-14T19:08:47.787 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2023-07-14T19:08:47.787 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /run 2023-07-14T19:08:47.787 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2023-07-14T19:08:47.787 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2023-07-14T19:08:47.787 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2023-07-14T19:08:47.787 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2023-07-14T19:08:47.787 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2023-07-14T19:08:47.788 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:08:48.144 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:08:49.188 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:08:49.362 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:08:49.581 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:08:49.730 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:08:49.760 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:08:49.912 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:08:50.131 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:08:50.215 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:08:50.310 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:08:50.394 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:08:50.411 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:08:50.763 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:08:50.944 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:08:50.961 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:08:52.047 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:08:52.223 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:08:52.341 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T19:08:52.524 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T19:08:52.701 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:08:52.754 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:08:52.917 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:08:52.947 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:08:53.074 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T19:08:53.203 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:08:53.282 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T19:08:53.500 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:08:53.714 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:08:54.011 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:08:54.162 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:08:54.301 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:08:54.392 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:08:54.747 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T19:08:54.948 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:08:54.972 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:08:54.983 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:08:55.127 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:08:55.136 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:08:55.168 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:08:55.259 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:08:55.630 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T19:08:55.880 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:08:55.914 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:08:56.059 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:08:56.070 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:08:56.422 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T19:08:56.600 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:08:56.635 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:08:56.784 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:08:56.826 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:08:56.975 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:08:57.326 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:08:57.366 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T19:08:57.484 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:08:57.582 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:08:57.617 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:08:57.759 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:08:57.800 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:08:57.959 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:08:58.311 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:08:58.459 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:09:02.036 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:09:02.903 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T19:09:02.996 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:09:03.205 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T19:09:03.301 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T19:09:03.377 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T19:09:03.449 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T19:09:03.629 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T19:09:03.637 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T19:09:03.807 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:09:04.001 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:09:04.220 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T19:09:04.394 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T19:09:04.533 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:09:04.554 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-14T19:09:04.880 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-14T19:09:05.085 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 119/134 2023-07-14T19:09:05.252 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T19:09:05.410 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T19:09:05.574 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:09:06.723 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:09:06.973 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:09:06.991 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:09:07.157 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T19:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T19:09:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T19:09:07.725 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:09:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T19:09:08.031 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:09:08.049 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:09:08.190 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T19:09:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:09:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:09:08.456 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:09:08.469 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:09:08.472 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T19:09:08.606 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:09:08.614 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:09:08.741 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T19:09:08.742 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:09:08.762 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T19:09:08.942 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:09:08.943 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:09:09.082 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T19:09:09.214 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T19:09:09.222 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T19:09:09.228 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:09:09.407 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-14T19:09:09.413 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:09:09.630 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T19:09:09.709 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T19:09:09.897 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T19:09:09.910 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T19:09:09.910 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:09:09.919 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:09:09.919 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T19:09:09.920 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T19:09:09.920 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T19:09:09.920 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T19:09:09.920 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T19:09:09.920 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:09:09.920 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:09:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:09:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:09:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T19:09:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T19:09:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T19:09:09.921 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T19:09:09.922 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T19:09:09.922 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T19:09:09.922 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T19:09:09.922 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T19:09:09.922 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T19:09:09.922 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T19:09:09.923 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T19:09:09.923 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T19:09:09.923 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T19:09:09.923 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T19:09:09.924 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T19:09:09.924 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T19:09:09.924 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T19:09:09.925 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T19:09:09.925 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T19:09:09.926 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T19:09:09.926 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T19:09:09.926 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T19:09:09.927 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T19:09:09.927 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T19:09:09.927 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T19:09:09.927 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T19:09:09.928 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T19:09:09.928 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T19:09:09.928 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T19:09:09.928 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T19:09:09.929 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T19:09:09.929 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T19:09:09.929 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T19:09:09.930 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T19:09:09.930 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T19:09:09.930 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T19:09:09.930 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T19:09:09.931 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T19:09:09.931 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T19:09:09.932 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T19:09:09.932 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T19:09:09.932 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T19:09:09.933 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T19:09:09.933 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T19:09:09.933 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T19:09:09.933 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T19:09:09.934 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T19:09:09.934 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T19:09:09.934 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T19:09:09.935 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T19:09:09.935 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T19:09:09.935 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T19:09:09.936 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T19:09:09.936 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T19:09:09.936 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T19:09:09.937 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T19:09:09.937 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T19:09:09.937 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T19:09:09.938 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T19:09:09.938 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T19:09:09.938 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T19:09:09.939 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T19:09:09.939 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T19:09:09.939 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T19:09:09.939 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T19:09:09.940 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T19:09:09.940 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T19:09:09.940 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T19:09:09.940 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T19:09:09.941 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T19:09:09.941 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T19:09:09.941 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T19:09:09.942 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T19:09:09.942 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T19:09:09.942 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T19:09:09.942 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T19:09:09.943 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T19:09:09.943 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T19:09:09.943 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T19:09:09.944 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T19:09:09.944 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T19:09:09.944 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T19:09:09.944 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T19:09:09.945 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T19:09:09.945 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T19:09:09.945 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T19:09:09.946 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T19:09:09.946 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T19:09:09.947 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T19:09:09.947 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T19:09:09.947 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T19:09:09.947 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T19:09:09.948 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T19:09:09.948 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T19:09:09.948 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T19:09:09.949 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T19:09:09.949 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T19:09:09.949 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T19:09:09.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T19:09:09.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T19:09:09.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T19:09:09.950 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T19:09:09.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T19:09:09.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T19:09:09.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T19:09:09.951 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T19:09:09.952 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T19:09:09.952 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T19:09:09.952 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T19:09:09.953 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T19:09:09.953 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T19:09:09.953 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T19:09:09.953 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T19:09:09.954 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T19:09:09.954 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T19:09:09.954 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T19:09:09.955 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T19:09:09.955 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T19:09:09.955 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T19:09:09.955 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T19:09:09.955 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T19:09:09.955 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T19:09:09.956 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:09:09.956 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T19:09:10.084 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:09:10.084 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:09:10.337 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:09:10.512 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:09:11.006 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:09:11.006 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T19:09:11.006 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T19:09:11.006 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T19:09:11.006 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T19:09:11.006 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T19:09:11.007 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:09:11.007 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:09:11.007 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:09:11.007 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:09:11.007 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T19:09:11.007 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T19:09:11.007 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T19:09:11.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T19:09:11.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T19:09:11.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T19:09:11.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T19:09:11.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T19:09:11.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T19:09:11.008 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T19:09:11.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T19:09:11.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T19:09:11.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T19:09:11.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T19:09:11.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T19:09:11.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T19:09:11.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T19:09:11.009 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T19:09:11.023 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T19:09:11.023 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T19:09:11.023 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T19:09:11.024 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T19:09:11.024 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T19:09:11.024 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T19:09:11.024 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T19:09:11.024 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T19:09:11.024 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T19:09:11.025 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T19:09:11.025 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T19:09:11.025 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T19:09:11.025 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T19:09:11.026 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T19:09:11.026 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T19:09:11.026 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T19:09:11.026 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T19:09:11.026 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T19:09:11.026 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T19:09:11.027 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T19:09:11.027 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T19:09:11.027 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T19:09:11.028 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T19:09:11.028 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T19:09:11.028 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T19:09:11.029 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T19:09:11.029 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T19:09:11.029 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T19:09:11.029 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T19:09:11.029 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T19:09:11.030 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T19:09:11.030 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T19:09:11.030 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T19:09:11.030 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T19:09:11.030 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T19:09:11.031 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T19:09:11.031 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T19:09:11.031 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T19:09:11.031 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T19:09:11.031 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T19:09:11.031 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T19:09:11.031 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T19:09:11.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T19:09:11.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T19:09:11.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-14T19:09:11.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-14T19:09:11.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-14T19:09:11.032 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-14T19:09:11.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-14T19:09:11.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-14T19:09:11.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-14T19:09:11.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-14T19:09:11.033 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-14T19:09:11.034 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-14T19:09:11.034 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-14T19:09:11.034 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-14T19:09:11.034 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-14T19:09:11.034 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-14T19:09:11.034 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-14T19:09:11.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-14T19:09:11.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-14T19:09:11.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-14T19:09:11.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-14T19:09:11.035 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-14T19:09:11.036 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-14T19:09:11.036 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-14T19:09:11.036 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-14T19:09:11.036 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-14T19:09:11.036 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-14T19:09:11.037 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-14T19:09:11.037 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-14T19:09:11.037 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-14T19:09:11.037 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-14T19:09:11.037 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-14T19:09:11.037 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-14T19:09:11.037 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-14T19:09:11.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-14T19:09:11.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-14T19:09:11.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-14T19:09:11.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-14T19:09:11.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-14T19:09:11.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-14T19:09:11.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-14T19:09:11.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-14T19:09:11.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-14T19:09:11.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-14T19:09:11.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-14T19:09:11.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-14T19:09:11.040 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-14T19:09:11.040 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-14T19:09:11.040 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-14T19:09:11.040 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-14T19:09:11.041 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T19:09:11.041 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-14T19:09:11.041 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:09:11.042 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-14T19:09:11.042 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-14T19:09:11.042 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-14T19:09:11.042 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-14T19:09:11.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-14T19:09:11.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T19:09:11.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T19:09:11.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T19:09:11.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T19:09:11.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:09:11.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T19:09:12.326 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:09:12.326 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:02:19 ago on Fri 14 Jul 2023 07:06:53 PM UTC. 2023-07-14T19:09:13.476 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:09:13.476 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:20 ago on Fri 14 Jul 2023 07:06:53 PM UTC. 2023-07-14T19:09:14.861 INFO:teuthology.orchestra.run.smithi070.stdout:Installed products updated. 2023-07-14T19:09:15.072 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:09:15.072 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2023-07-14T19:09:15.073 INFO:teuthology.orchestra.run.smithi070.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T19:09:15.073 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.073 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.073 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2023-07-14T19:09:15.073 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T19:09:15.073 INFO:teuthology.orchestra.run.smithi070.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T19:09:15.073 INFO:teuthology.orchestra.run.smithi070.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T19:09:15.074 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.074 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.074 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.074 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.074 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.074 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.074 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.075 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.075 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.075 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.075 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.075 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.075 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.075 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.075 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.076 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.076 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.076 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.076 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.076 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.076 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.076 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.077 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:15.077 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T19:09:15.077 INFO:teuthology.orchestra.run.smithi070.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T19:09:15.077 INFO:teuthology.orchestra.run.smithi070.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T19:09:15.077 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T19:09:15.077 INFO:teuthology.orchestra.run.smithi070.stdout: jq-1.6-3.el8.x86_64 2023-07-14T19:09:15.077 INFO:teuthology.orchestra.run.smithi070.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T19:09:15.078 INFO:teuthology.orchestra.run.smithi070.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T19:09:15.078 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T19:09:15.078 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.078 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.078 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.078 INFO:teuthology.orchestra.run.smithi070.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T19:09:15.078 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:09:15.078 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T19:09:15.079 INFO:teuthology.orchestra.run.smithi070.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T19:09:15.079 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:09:15.079 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T19:09:15.079 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.079 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.079 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T19:09:15.079 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.079 INFO:teuthology.orchestra.run.smithi070.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T19:09:15.080 INFO:teuthology.orchestra.run.smithi070.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T19:09:15.080 INFO:teuthology.orchestra.run.smithi070.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T19:09:15.080 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T19:09:15.080 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T19:09:15.080 INFO:teuthology.orchestra.run.smithi070.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T19:09:15.080 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T19:09:15.080 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T19:09:15.081 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T19:09:15.081 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T19:09:15.081 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T19:09:15.081 INFO:teuthology.orchestra.run.smithi070.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T19:09:15.081 INFO:teuthology.orchestra.run.smithi070.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T19:09:15.081 INFO:teuthology.orchestra.run.smithi070.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T19:09:15.081 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:09:15.081 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:09:15.082 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:09:15.082 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T19:09:15.082 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T19:09:15.082 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T19:09:15.082 INFO:teuthology.orchestra.run.smithi070.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T19:09:15.082 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T19:09:15.082 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.082 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T19:09:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T19:09:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T19:09:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T19:09:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T19:09:15.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T19:09:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T19:09:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T19:09:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T19:09:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T19:09:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T19:09:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T19:09:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T19:09:15.084 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T19:09:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T19:09:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T19:09:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T19:09:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T19:09:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T19:09:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T19:09:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T19:09:15.085 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T19:09:15.086 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T19:09:15.086 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T19:09:15.086 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T19:09:15.086 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T19:09:15.086 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T19:09:15.086 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T19:09:15.086 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T19:09:15.087 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.087 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.087 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T19:09:15.087 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T19:09:15.087 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.087 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T19:09:15.087 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T19:09:15.087 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T19:09:15.088 INFO:teuthology.orchestra.run.smithi070.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T19:09:15.088 INFO:teuthology.orchestra.run.smithi070.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T19:09:15.088 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T19:09:15.088 INFO:teuthology.orchestra.run.smithi070.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T19:09:15.088 INFO:teuthology.orchestra.run.smithi070.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T19:09:15.088 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T19:09:15.088 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T19:09:15.088 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T19:09:15.089 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T19:09:15.089 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T19:09:15.089 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.089 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.089 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:15.089 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T19:09:15.089 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T19:09:15.089 INFO:teuthology.orchestra.run.smithi070.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T19:09:15.090 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T19:09:15.090 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T19:09:15.090 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T19:09:15.090 INFO:teuthology.orchestra.run.smithi070.stdout: 2023-07-14T19:09:15.091 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2023-07-14T19:09:15.281 DEBUG:teuthology.parallel:result is None 2023-07-14T19:09:15.973 INFO:teuthology.orchestra.run.smithi104.stdout:Installed products updated. 2023-07-14T19:09:16.187 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:09:16.187 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2023-07-14T19:09:16.188 INFO:teuthology.orchestra.run.smithi104.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T19:09:16.188 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.188 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.188 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2023-07-14T19:09:16.188 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T19:09:16.189 INFO:teuthology.orchestra.run.smithi104.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T19:09:16.189 INFO:teuthology.orchestra.run.smithi104.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T19:09:16.189 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.189 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.189 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.189 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.189 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.190 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.190 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.190 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.190 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.190 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.190 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.190 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.191 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.191 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.191 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.191 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.191 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.191 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.191 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.192 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.192 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.192 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.192 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:09:16.192 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T19:09:16.192 INFO:teuthology.orchestra.run.smithi104.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T19:09:16.192 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T19:09:16.193 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T19:09:16.193 INFO:teuthology.orchestra.run.smithi104.stdout: jq-1.6-3.el8.x86_64 2023-07-14T19:09:16.193 INFO:teuthology.orchestra.run.smithi104.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T19:09:16.193 INFO:teuthology.orchestra.run.smithi104.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T19:09:16.193 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T19:09:16.193 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.193 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.194 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.194 INFO:teuthology.orchestra.run.smithi104.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T19:09:16.194 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:09:16.194 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T19:09:16.194 INFO:teuthology.orchestra.run.smithi104.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T19:09:16.194 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:09:16.194 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T19:09:16.195 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.195 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.195 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T19:09:16.195 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.195 INFO:teuthology.orchestra.run.smithi104.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T19:09:16.195 INFO:teuthology.orchestra.run.smithi104.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T19:09:16.195 INFO:teuthology.orchestra.run.smithi104.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T19:09:16.196 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T19:09:16.196 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T19:09:16.196 INFO:teuthology.orchestra.run.smithi104.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T19:09:16.196 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T19:09:16.196 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T19:09:16.196 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T19:09:16.196 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T19:09:16.197 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T19:09:16.197 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T19:09:16.197 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T19:09:16.197 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T19:09:16.197 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:09:16.197 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:09:16.198 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:09:16.198 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T19:09:16.198 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T19:09:16.198 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T19:09:16.198 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T19:09:16.198 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T19:09:16.199 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.199 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.199 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.199 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T19:09:16.199 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T19:09:16.199 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T19:09:16.199 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T19:09:16.200 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T19:09:16.200 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T19:09:16.200 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T19:09:16.200 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T19:09:16.200 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T19:09:16.200 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T19:09:16.200 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T19:09:16.201 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T19:09:16.201 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T19:09:16.201 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T19:09:16.201 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T19:09:16.201 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T19:09:16.201 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T19:09:16.201 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T19:09:16.202 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T19:09:16.202 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T19:09:16.202 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T19:09:16.202 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T19:09:16.202 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T19:09:16.202 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T19:09:16.202 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T19:09:16.203 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T19:09:16.203 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T19:09:16.203 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T19:09:16.203 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T19:09:16.203 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.203 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.203 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T19:09:16.204 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T19:09:16.204 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.204 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T19:09:16.204 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T19:09:16.204 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T19:09:16.204 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T19:09:16.204 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T19:09:16.205 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T19:09:16.205 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T19:09:16.205 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T19:09:16.205 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T19:09:16.205 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T19:09:16.205 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T19:09:16.205 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T19:09:16.206 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T19:09:16.206 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.206 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.206 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:09:16.206 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T19:09:16.206 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T19:09:16.206 INFO:teuthology.orchestra.run.smithi104.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T19:09:16.207 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T19:09:16.207 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T19:09:16.207 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T19:09:16.207 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:09:16.208 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2023-07-14T19:09:16.365 DEBUG:teuthology.parallel:result is None 2023-07-14T19:11:16.926 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 58 kB/s | 16 MB 04:37 2023-07-14T19:11:20.592 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 348 kB/s | 24 kB 00:00 2023-07-14T19:11:21.511 INFO:teuthology.orchestra.run.smithi053.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 MB/s | 53 MB 00:00 2023-07-14T19:11:30.278 INFO:teuthology.orchestra.run.smithi053.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 59 MB/s | 47 MB 00:00 2023-07-14T19:11:41.945 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:11:41.946 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:11:41.961 INFO:teuthology.orchestra.run.smithi053.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T19:11:41.962 INFO:teuthology.orchestra.run.smithi053.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T19:11:41.963 INFO:teuthology.orchestra.run.smithi053.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T19:11:41.964 INFO:teuthology.orchestra.run.smithi053.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T19:11:41.966 INFO:teuthology.orchestra.run.smithi053.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T19:11:41.970 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T19:11:41.971 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T19:11:42.116 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-07-14T19:11:42.129 INFO:teuthology.orchestra.run.smithi053.stdout:=========================================================================================================================== 2023-07-14T19:11:42.129 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-07-14T19:11:42.129 INFO:teuthology.orchestra.run.smithi053.stdout:=========================================================================================================================== 2023-07-14T19:11:42.129 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-07-14T19:11:42.130 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T19:11:42.130 INFO:teuthology.orchestra.run.smithi053.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T19:11:42.130 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T19:11:42.130 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T19:11:42.130 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T19:11:42.130 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T19:11:42.130 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T19:11:42.131 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T19:11:42.131 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T19:11:42.131 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T19:11:42.131 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T19:11:42.131 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T19:11:42.131 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T19:11:42.131 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T19:11:42.132 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T19:11:42.132 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T19:11:42.132 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T19:11:42.132 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T19:11:42.132 INFO:teuthology.orchestra.run.smithi053.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T19:11:42.132 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T19:11:42.132 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T19:11:42.132 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T19:11:42.133 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T19:11:42.133 INFO:teuthology.orchestra.run.smithi053.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T19:11:42.133 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T19:11:42.133 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T19:11:42.133 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T19:11:42.133 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T19:11:42.133 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T19:11:42.134 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T19:11:42.134 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T19:11:42.134 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T19:11:42.134 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T19:11:42.134 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T19:11:42.134 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T19:11:42.134 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T19:11:42.134 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T19:11:42.135 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2023-07-14T19:11:42.135 INFO:teuthology.orchestra.run.smithi053.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T19:11:42.135 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T19:11:42.135 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T19:11:42.135 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-07-14T19:11:42.135 INFO:teuthology.orchestra.run.smithi053.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T19:11:42.135 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T19:11:42.136 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T19:11:42.136 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T19:11:42.136 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T19:11:42.136 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T19:11:42.136 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T19:11:42.136 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T19:11:42.137 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T19:11:42.137 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T19:11:42.137 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T19:11:42.137 INFO:teuthology.orchestra.run.smithi053.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T19:11:42.137 INFO:teuthology.orchestra.run.smithi053.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T19:11:42.137 INFO:teuthology.orchestra.run.smithi053.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T19:11:42.137 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T19:11:42.138 INFO:teuthology.orchestra.run.smithi053.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T19:11:42.138 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T19:11:42.138 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T19:11:42.138 INFO:teuthology.orchestra.run.smithi053.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T19:11:42.138 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T19:11:42.138 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T19:11:42.138 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T19:11:42.139 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T19:11:42.139 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T19:11:42.139 INFO:teuthology.orchestra.run.smithi053.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T19:11:42.139 INFO:teuthology.orchestra.run.smithi053.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T19:11:42.139 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T19:11:42.139 INFO:teuthology.orchestra.run.smithi053.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T19:11:42.140 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T19:11:42.140 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T19:11:42.140 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T19:11:42.140 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T19:11:42.140 INFO:teuthology.orchestra.run.smithi053.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T19:11:42.140 INFO:teuthology.orchestra.run.smithi053.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T19:11:42.141 INFO:teuthology.orchestra.run.smithi053.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T19:11:42.141 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T19:11:42.141 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T19:11:42.141 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T19:11:42.141 INFO:teuthology.orchestra.run.smithi053.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T19:11:42.141 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T19:11:42.142 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T19:11:42.142 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T19:11:42.142 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T19:11:42.142 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T19:11:42.142 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T19:11:42.142 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T19:11:42.142 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T19:11:42.143 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T19:11:42.143 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T19:11:42.143 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T19:11:42.143 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T19:11:42.143 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T19:11:42.143 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T19:11:42.144 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:11:42.144 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T19:11:42.144 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T19:11:42.144 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T19:11:42.144 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T19:11:42.145 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T19:11:42.145 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T19:11:42.145 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T19:11:42.145 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T19:11:42.145 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T19:11:42.146 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T19:11:42.146 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T19:11:42.146 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T19:11:42.146 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T19:11:42.146 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T19:11:42.146 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T19:11:42.146 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T19:11:42.146 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T19:11:42.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T19:11:42.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:11:42.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T19:11:42.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T19:11:42.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T19:11:42.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T19:11:42.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T19:11:42.148 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T19:11:42.148 INFO:teuthology.orchestra.run.smithi053.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T19:11:42.148 INFO:teuthology.orchestra.run.smithi053.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T19:11:42.148 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T19:11:42.148 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T19:11:42.148 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T19:11:42.148 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T19:11:42.148 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T19:11:42.149 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T19:11:42.149 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T19:11:42.149 INFO:teuthology.orchestra.run.smithi053.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T19:11:42.149 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T19:11:42.149 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T19:11:42.149 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling module streams: 2023-07-14T19:11:42.149 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql 10 2023-07-14T19:11:42.150 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:11:42.150 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-07-14T19:11:42.150 INFO:teuthology.orchestra.run.smithi053.stdout:=========================================================================================================================== 2023-07-14T19:11:42.150 INFO:teuthology.orchestra.run.smithi053.stdout:Install 128 Packages 2023-07-14T19:11:42.150 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 3 Packages 2023-07-14T19:11:42.150 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:11:42.151 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 206 M 2023-07-14T19:11:42.151 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-07-14T19:11:43.304 INFO:teuthology.orchestra.run.smithi053.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 14 kB/s | 6.4 kB 00:00 2023-07-14T19:11:44.712 INFO:teuthology.orchestra.run.smithi053.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 604 kB/s | 850 kB 00:01 2023-07-14T19:11:44.962 INFO:teuthology.orchestra.run.smithi053.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 668 kB/s | 167 kB 00:00 2023-07-14T19:11:47.737 INFO:teuthology.orchestra.run.smithi053.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 847 kB/s | 2.3 MB 00:02 2023-07-14T19:11:50.013 INFO:teuthology.orchestra.run.smithi053.stdout:(5/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 767 kB/s | 5.4 MB 00:07 2023-07-14T19:11:50.338 INFO:teuthology.orchestra.run.smithi053.stdout:(6/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 611 kB/s | 1.6 MB 00:02 2023-07-14T19:11:55.996 INFO:teuthology.orchestra.run.smithi053.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 790 kB/s | 4.6 MB 00:05 2023-07-14T19:12:18.578 INFO:teuthology.orchestra.run.smithi053.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 635 kB/s | 22 MB 00:35 2023-07-14T19:12:18.869 INFO:teuthology.orchestra.run.smithi053.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 83 kB/s | 24 kB 00:00 2023-07-14T19:12:19.453 INFO:teuthology.orchestra.run.smithi053.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 591 kB/s | 14 MB 00:23 2023-07-14T19:12:19.687 INFO:teuthology.orchestra.run.smithi053.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 1.1 MB/s | 260 kB 00:00 2023-07-14T19:12:19.829 INFO:teuthology.orchestra.run.smithi053.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 219 kB/s | 31 kB 00:00 2023-07-14T19:12:20.271 INFO:teuthology.orchestra.run.smithi053.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 1.7 MB/s | 772 kB 00:00 2023-07-14T19:12:20.412 INFO:teuthology.orchestra.run.smithi053.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 957 kB/s | 135 kB 00:00 2023-07-14T19:12:20.604 INFO:teuthology.orchestra.run.smithi053.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 725 kB/s | 139 kB 00:00 2023-07-14T19:12:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:(16/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 597 kB/s | 18 MB 00:30 2023-07-14T19:12:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:(17/131): libradosstriper1-18.0.0-4940.g81c9387 1.3 MB/s | 473 kB 00:00 2023-07-14T19:12:21.147 INFO:teuthology.orchestra.run.smithi053.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 256 kB/s | 45 kB 00:00 2023-07-14T19:12:21.314 INFO:teuthology.orchestra.run.smithi053.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 735 kB/s | 122 kB 00:00 2023-07-14T19:12:21.606 INFO:teuthology.orchestra.run.smithi053.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 675 kB/s | 197 kB 00:00 2023-07-14T19:12:22.081 INFO:teuthology.orchestra.run.smithi053.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 781 kB/s | 371 kB 00:00 2023-07-14T19:12:23.065 INFO:teuthology.orchestra.run.smithi053.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 370 kB/s | 364 kB 00:00 2023-07-14T19:12:24.006 INFO:teuthology.orchestra.run.smithi053.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 113 kB/s | 107 kB 00:00 2023-07-14T19:12:25.065 INFO:teuthology.orchestra.run.smithi053.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 88 kB/s | 94 kB 00:01 2023-07-14T19:12:36.822 INFO:teuthology.orchestra.run.smithi053.stdout:(25/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 308 kB/s | 3.5 MB 00:11 2023-07-14T19:12:37.139 INFO:teuthology.orchestra.run.smithi053.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 565 kB/s | 179 kB 00:00 2023-07-14T19:12:37.256 INFO:teuthology.orchestra.run.smithi053.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 198 kB/s | 23 kB 00:00 2023-07-14T19:12:37.481 INFO:teuthology.orchestra.run.smithi053.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 575 kB/s | 129 kB 00:00 2023-07-14T19:12:37.573 INFO:teuthology.orchestra.run.smithi053.stdout:(29/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 414 kB/s | 6.8 MB 00:16 2023-07-14T19:12:38.848 INFO:teuthology.orchestra.run.smithi053.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.3 MB/s | 1.7 MB 00:01 2023-07-14T19:12:39.065 INFO:teuthology.orchestra.run.smithi053.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 1.1 MB/s | 241 kB 00:00 2023-07-14T19:12:39.174 INFO:teuthology.orchestra.run.smithi053.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 437 kB/s | 47 kB 00:00 2023-07-14T19:12:39.282 INFO:teuthology.orchestra.run.smithi053.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 134 kB/s | 15 kB 00:00 2023-07-14T19:12:39.491 INFO:teuthology.orchestra.run.smithi053.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 1.2 MB/s | 258 kB 00:00 2023-07-14T19:12:39.608 INFO:teuthology.orchestra.run.smithi053.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 1.4 MB/s | 165 kB 00:00 2023-07-14T19:12:39.716 INFO:teuthology.orchestra.run.smithi053.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 166 kB/s | 18 kB 00:00 2023-07-14T19:12:40.725 INFO:teuthology.orchestra.run.smithi053.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 304 kB/s | 306 kB 00:01 2023-07-14T19:12:40.834 INFO:teuthology.orchestra.run.smithi053.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 542 kB/s | 59 kB 00:00 2023-07-14T19:12:40.917 INFO:teuthology.orchestra.run.smithi053.stdout:(39/131): ceph-mgr-diskprediction-local-18.0.0- 2.2 MB/s | 7.4 MB 00:03 2023-07-14T19:12:40.934 INFO:teuthology.orchestra.run.smithi053.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 749 kB/s | 75 kB 00:00 2023-07-14T19:12:41.043 INFO:teuthology.orchestra.run.smithi053.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 403 kB/s | 44 kB 00:00 2023-07-14T19:12:41.176 INFO:teuthology.orchestra.run.smithi053.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2023-07-14T19:12:41.477 INFO:teuthology.orchestra.run.smithi053.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 110 kB/s | 33 kB 00:00 2023-07-14T19:12:41.594 INFO:teuthology.orchestra.run.smithi053.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 140 kB/s | 16 kB 00:00 2023-07-14T19:12:42.077 INFO:teuthology.orchestra.run.smithi053.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 357 kB/s | 173 kB 00:00 2023-07-14T19:12:42.453 INFO:teuthology.orchestra.run.smithi053.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.0 MB/s | 384 kB 00:00 2023-07-14T19:12:42.561 INFO:teuthology.orchestra.run.smithi053.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 749 kB/s | 81 kB 00:00 2023-07-14T19:12:42.695 INFO:teuthology.orchestra.run.smithi053.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-07-14T19:12:42.820 INFO:teuthology.orchestra.run.smithi053.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 141 kB/s | 18 kB 00:00 2023-07-14T19:12:43.929 INFO:teuthology.orchestra.run.smithi053.stdout:(50/131): python3-asyncssh-2.7.0-2.el8.noarch.r 158 kB/s | 476 kB 00:03 2023-07-14T19:12:44.287 INFO:teuthology.orchestra.run.smithi053.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 136 kB/s | 49 kB 00:00 2023-07-14T19:12:44.771 INFO:teuthology.orchestra.run.smithi053.stdout:(52/131): python3-kubernetes-11.0.0-6.el8.noarc 612 kB/s | 1.2 MB 00:01 2023-07-14T19:12:44.788 INFO:teuthology.orchestra.run.smithi053.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 117 kB/s | 59 kB 00:00 2023-07-14T19:12:44.922 INFO:teuthology.orchestra.run.smithi053.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 401 kB/s | 60 kB 00:00 2023-07-14T19:12:45.055 INFO:teuthology.orchestra.run.smithi053.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 117 kB/s | 16 kB 00:00 2023-07-14T19:12:45.197 INFO:teuthology.orchestra.run.smithi053.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 238 kB/s | 34 kB 00:00 2023-07-14T19:12:45.572 INFO:teuthology.orchestra.run.smithi053.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 524 kB/s | 196 kB 00:00 2023-07-14T19:12:45.706 INFO:teuthology.orchestra.run.smithi053.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 462 kB/s | 61 kB 00:00 2023-07-14T19:12:45.848 INFO:teuthology.orchestra.run.smithi053.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 132 kB/s | 19 kB 00:00 2023-07-14T19:12:45.982 INFO:teuthology.orchestra.run.smithi053.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 179 kB/s | 24 kB 00:00 2023-07-14T19:12:46.132 INFO:teuthology.orchestra.run.smithi053.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 195 kB/s | 29 kB 00:00 2023-07-14T19:12:46.282 INFO:teuthology.orchestra.run.smithi053.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 184 kB/s | 27 kB 00:00 2023-07-14T19:12:46.357 INFO:teuthology.orchestra.run.smithi053.stdout:(63/131): python3-pecan-1.3.2-9.el8.noarch.rpm 180 kB/s | 283 kB 00:01 2023-07-14T19:12:47.025 INFO:teuthology.orchestra.run.smithi053.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 344 kB/s | 255 kB 00:00 2023-07-14T19:12:47.300 INFO:teuthology.orchestra.run.smithi053.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 222 kB/s | 61 kB 00:00 2023-07-14T19:12:47.433 INFO:teuthology.orchestra.run.smithi053.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 233 kB/s | 251 kB 00:01 2023-07-14T19:12:47.534 INFO:teuthology.orchestra.run.smithi053.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 365 kB/s | 85 kB 00:00 2023-07-14T19:12:47.559 INFO:teuthology.orchestra.run.smithi053.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 183 kB/s | 23 kB 00:00 2023-07-14T19:12:47.584 INFO:teuthology.orchestra.run.smithi053.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T19:12:47.660 INFO:teuthology.orchestra.run.smithi053.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 626 kB/s | 47 kB 00:00 2023-07-14T19:12:47.710 INFO:teuthology.orchestra.run.smithi053.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 3.9 MB/s | 200 kB 00:00 2023-07-14T19:12:47.760 INFO:teuthology.orchestra.run.smithi053.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 853 kB/s | 43 kB 00:00 2023-07-14T19:12:47.794 INFO:teuthology.orchestra.run.smithi053.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 6.5 MB/s | 220 kB 00:00 2023-07-14T19:12:47.836 INFO:teuthology.orchestra.run.smithi053.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.1 MB/s | 48 kB 00:00 2023-07-14T19:12:47.878 INFO:teuthology.orchestra.run.smithi053.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-07-14T19:12:47.920 INFO:teuthology.orchestra.run.smithi053.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2023-07-14T19:12:47.962 INFO:teuthology.orchestra.run.smithi053.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 2.6 MB/s | 109 kB 00:00 2023-07-14T19:12:48.003 INFO:teuthology.orchestra.run.smithi053.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-07-14T19:12:48.054 INFO:teuthology.orchestra.run.smithi053.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 11 MB/s | 560 kB 00:00 2023-07-14T19:12:48.096 INFO:teuthology.orchestra.run.smithi053.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 451 kB/s | 19 kB 00:00 2023-07-14T19:12:48.154 INFO:teuthology.orchestra.run.smithi053.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 2.6 MB/s | 155 kB 00:00 2023-07-14T19:12:48.205 INFO:teuthology.orchestra.run.smithi053.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 11 MB/s | 559 kB 00:00 2023-07-14T19:12:48.247 INFO:teuthology.orchestra.run.smithi053.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 429 kB/s | 18 kB 00:00 2023-07-14T19:12:48.305 INFO:teuthology.orchestra.run.smithi053.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2023-07-14T19:12:48.355 INFO:teuthology.orchestra.run.smithi053.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.7 MB/s | 238 kB 00:00 2023-07-14T19:12:48.439 INFO:teuthology.orchestra.run.smithi053.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 28 MB/s | 2.3 MB 00:00 2023-07-14T19:12:48.481 INFO:teuthology.orchestra.run.smithi053.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-07-14T19:12:48.525 INFO:teuthology.orchestra.run.smithi053.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 2.0 MB/s | 83 kB 00:00 2023-07-14T19:12:48.565 INFO:teuthology.orchestra.run.smithi053.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2023-07-14T19:12:48.707 INFO:teuthology.orchestra.run.smithi053.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 695 kB/s | 98 kB 00:00 2023-07-14T19:12:48.766 INFO:teuthology.orchestra.run.smithi053.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 989 kB/s | 58 kB 00:00 2023-07-14T19:12:48.808 INFO:teuthology.orchestra.run.smithi053.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 929 kB/s | 39 kB 00:00 2023-07-14T19:12:48.883 INFO:teuthology.orchestra.run.smithi053.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 8.4 MB/s | 644 kB 00:00 2023-07-14T19:12:48.925 INFO:teuthology.orchestra.run.smithi053.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 4.0 MB/s | 171 kB 00:00 2023-07-14T19:12:48.967 INFO:teuthology.orchestra.run.smithi053.stdout:(95/131): NetworkManager-initscripts-updown-1.3 3.2 MB/s | 138 kB 00:00 2023-07-14T19:12:49.268 INFO:teuthology.orchestra.run.smithi053.stdout:(96/131): python3-scipy-1.0.0-21.module+el8.5.0 46 MB/s | 14 MB 00:00 2023-07-14T19:12:49.309 INFO:teuthology.orchestra.run.smithi053.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-07-14T19:12:49.351 INFO:teuthology.orchestra.run.smithi053.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-07-14T19:12:49.393 INFO:teuthology.orchestra.run.smithi053.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-14T19:12:49.444 INFO:teuthology.orchestra.run.smithi053.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-07-14T19:12:49.486 INFO:teuthology.orchestra.run.smithi053.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 931 kB/s | 39 kB 00:00 2023-07-14T19:12:49.552 INFO:teuthology.orchestra.run.smithi053.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2023-07-14T19:12:49.603 INFO:teuthology.orchestra.run.smithi053.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 11 MB/s | 538 kB 00:00 2023-07-14T19:12:49.645 INFO:teuthology.orchestra.run.smithi053.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-07-14T19:12:49.695 INFO:teuthology.orchestra.run.smithi053.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-07-14T19:12:49.754 INFO:teuthology.orchestra.run.smithi053.stdout:(106/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-07-14T19:12:49.987 INFO:teuthology.orchestra.run.smithi053.stdout:(107/131): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-07-14T19:12:50.054 INFO:teuthology.orchestra.run.smithi053.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 399 kB/s | 27 kB 00:00 2023-07-14T19:12:50.105 INFO:teuthology.orchestra.run.smithi053.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-07-14T19:12:50.146 INFO:teuthology.orchestra.run.smithi053.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.0 MB/s | 44 kB 00:00 2023-07-14T19:12:50.272 INFO:teuthology.orchestra.run.smithi053.stdout:(111/131): python3-numpy-1.14.3-10.el8.x86_64.r 30 MB/s | 3.7 MB 00:00 2023-07-14T19:12:50.322 INFO:teuthology.orchestra.run.smithi053.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-07-14T19:12:50.372 INFO:teuthology.orchestra.run.smithi053.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 4.4 MB/s | 225 kB 00:00 2023-07-14T19:12:50.414 INFO:teuthology.orchestra.run.smithi053.stdout:(114/131): uuid-1.6.2-43.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2023-07-14T19:12:50.456 INFO:teuthology.orchestra.run.smithi053.stdout:(115/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 3.0 MB/s | 126 kB 00:00 2023-07-14T19:12:50.498 INFO:teuthology.orchestra.run.smithi053.stdout:(116/131): python3-jsonpointer-1.10-11.el8.noar 479 kB/s | 20 kB 00:00 2023-07-14T19:12:50.532 INFO:teuthology.orchestra.run.smithi053.stdout:(117/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 3.0 MB/s | 103 kB 00:00 2023-07-14T19:12:50.582 INFO:teuthology.orchestra.run.smithi053.stdout:(118/131): libtool-2.4.6-25.el8.x86_64.rpm 14 MB/s | 709 kB 00:00 2023-07-14T19:12:50.716 INFO:teuthology.orchestra.run.smithi053.stdout:(119/131): openblas-threads-0.3.15-3.el8.x86_64 36 MB/s | 4.9 MB 00:00 2023-07-14T19:12:50.749 INFO:teuthology.orchestra.run.smithi053.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 5.9 MB/s | 198 kB 00:00 2023-07-14T19:12:50.875 INFO:teuthology.orchestra.run.smithi053.stdout:(121/131): openblas-0.3.15-3.el8.x86_64.rpm 38 MB/s | 4.7 MB 00:00 2023-07-14T19:12:50.917 INFO:teuthology.orchestra.run.smithi053.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-07-14T19:12:50.959 INFO:teuthology.orchestra.run.smithi053.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-07-14T19:12:51.084 INFO:teuthology.orchestra.run.smithi053.stdout:(124/131): postgresql-server-10.21-2.module+el8 41 MB/s | 5.1 MB 00:00 2023-07-14T19:12:51.143 INFO:teuthology.orchestra.run.smithi053.stdout:(125/131): postgresql-contrib-10.21-2.module+el 14 MB/s | 808 kB 00:00 2023-07-14T19:12:51.201 INFO:teuthology.orchestra.run.smithi053.stdout:(126/131): postgresql-10.21-2.module+el8.6.0+15 26 MB/s | 1.5 MB 00:00 2023-07-14T19:12:52.160 INFO:teuthology.orchestra.run.smithi053.stdout:(127/131): thrift-0.13.0-2.el8.x86_64.rpm 384 kB/s | 1.7 MB 00:04 2023-07-14T19:12:53.194 INFO:teuthology.orchestra.run.smithi053.stdout:(128/131): librados2-18.0.0-4940.g81c9387d.el8. 1.8 MB/s | 3.7 MB 00:01 2023-07-14T19:12:53.252 INFO:teuthology.orchestra.run.smithi053.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 14 MB/s | 798 kB 00:00 2023-07-14T19:12:54.177 INFO:teuthology.orchestra.run.smithi053.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.8 MB/s | 3.6 MB 00:02 2023-07-14T19:13:31.932 INFO:teuthology.orchestra.run.smithi053.stdout:(131/131): ceph-test-18.0.0-4940.g81c9387d.el8. 672 kB/s | 48 MB 01:13 2023-07-14T19:13:31.935 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:13:31.935 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.9 MB/s | 206 MB 01:49 2023-07-14T19:13:32.737 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-07-14T19:13:32.871 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-07-14T19:13:32.871 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-07-14T19:13:34.046 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-07-14T19:13:34.051 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-07-14T19:13:38.990 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-07-14T19:13:39.176 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T19:13:39.370 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T19:13:39.561 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:13:39.668 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:13:39.878 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:13:40.322 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:13:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:13:40.949 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:13:41.120 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:13:41.253 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:13:41.453 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:13:41.557 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:13:41.725 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:13:41.913 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T19:13:42.154 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T19:13:42.373 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T19:13:42.795 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T19:13:43.324 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T19:13:43.469 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:13:43.570 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:13:43.728 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:13:43.906 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:13:44.116 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T19:13:44.317 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T19:13:44.503 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T19:13:44.636 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:13:44.774 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:13:44.919 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:13:45.573 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:13:45.736 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:13:46.370 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:13:46.544 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:13:47.407 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:13:47.659 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T19:13:47.855 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T19:13:48.192 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T19:13:48.359 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T19:13:48.490 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T19:13:48.722 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T19:13:48.935 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:13:50.830 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:13:51.893 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T19:13:52.200 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T19:13:52.526 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T19:13:52.727 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T19:13:52.933 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T19:13:53.123 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T19:13:53.300 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T19:13:53.459 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T19:13:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T19:13:53.738 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T19:13:53.909 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:13:53.969 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:13:54.221 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T19:13:54.432 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T19:13:54.568 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:13:54.648 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:13:54.828 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T19:13:54.964 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T19:13:55.119 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T19:13:55.280 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T19:13:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T19:13:55.631 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T19:13:55.878 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T19:13:56.095 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T19:13:56.508 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T19:13:56.804 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T19:13:57.031 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T19:13:57.256 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T19:13:57.484 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T19:13:57.811 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T19:13:58.043 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T19:13:58.254 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T19:13:58.495 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T19:13:58.683 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T19:13:58.848 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T19:13:59.151 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T19:13:59.419 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T19:13:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T19:13:59.903 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T19:14:00.059 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T19:14:00.231 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T19:14:00.509 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T19:14:00.725 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T19:14:00.855 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T19:14:00.996 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T19:14:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T19:14:01.399 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T19:14:01.581 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T19:14:01.779 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T19:14:01.915 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T19:14:02.145 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T19:14:02.364 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T19:14:02.548 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T19:14:02.780 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T19:14:03.042 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T19:14:03.184 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T19:14:03.388 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T19:14:04.549 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T19:14:04.824 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T19:14:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T19:14:05.373 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T19:14:05.662 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T19:14:05.844 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T19:14:06.651 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T19:14:06.883 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:14:06.952 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:14:07.142 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T19:14:11.027 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:14:11.224 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:14:11.839 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:14:12.146 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:14:12.231 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:14:12.374 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:14:31.788 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:14:31.789 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2023-07-14T19:14:31.789 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2023-07-14T19:14:31.789 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /dev 2023-07-14T19:14:31.790 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /run 2023-07-14T19:14:31.790 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2023-07-14T19:14:31.790 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2023-07-14T19:14:31.790 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2023-07-14T19:14:31.791 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2023-07-14T19:14:31.791 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /dev 2023-07-14T19:14:31.791 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:14:32.113 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:14:33.798 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:14:33.976 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:14:34.208 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:14:34.371 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:14:34.854 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:14:35.013 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:14:35.030 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:14:36.754 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:14:36.903 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:14:37.027 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T19:14:37.213 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T19:14:37.387 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:14:37.609 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:14:37.853 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:14:38.875 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:14:39.037 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:14:39.134 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:14:39.504 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T19:14:39.715 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:14:39.751 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:14:39.887 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:14:39.898 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:14:41.296 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T19:14:41.474 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:14:41.510 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:14:41.654 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:14:41.697 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:14:41.887 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:14:42.260 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:14:42.412 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:14:47.269 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:14:48.194 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T19:14:48.531 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T19:14:48.706 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T19:14:48.971 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T19:14:49.119 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:14:49.334 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:14:49.590 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T19:14:49.765 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T19:14:49.920 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:14:52.410 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:14:52.644 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:14:52.663 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:14:52.811 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T19:14:53.094 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T19:14:53.397 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T19:14:53.762 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T19:14:53.943 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:14:54.032 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:14:54.235 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:14:54.248 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:14:54.402 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:14:54.410 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:14:54.537 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T19:14:54.537 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:14:54.755 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:14:54.755 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:14:55.007 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:14:55.177 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:14:55.708 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:14:55.708 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T19:14:55.708 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T19:14:55.708 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T19:14:55.708 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T19:14:55.709 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T19:14:55.709 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:14:55.709 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:14:55.709 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:14:55.709 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:14:55.709 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T19:14:55.709 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T19:14:55.710 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T19:14:55.710 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T19:14:55.710 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T19:14:55.710 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T19:14:55.710 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T19:14:55.710 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T19:14:55.710 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T19:14:55.710 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T19:14:55.711 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T19:14:55.711 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T19:14:55.711 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T19:14:55.711 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T19:14:55.711 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T19:14:55.711 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T19:14:55.711 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T19:14:55.711 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T19:14:55.712 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T19:14:55.712 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T19:14:55.722 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T19:14:55.723 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T19:14:55.723 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T19:14:55.723 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T19:14:55.723 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T19:14:55.724 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T19:14:55.724 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T19:14:55.724 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T19:14:55.724 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T19:14:55.724 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T19:14:55.724 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T19:14:55.725 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T19:14:55.725 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T19:14:55.725 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T19:14:55.725 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T19:14:55.725 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T19:14:55.726 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T19:14:55.726 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T19:14:55.726 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T19:14:55.726 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T19:14:55.726 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T19:14:55.726 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T19:14:55.727 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T19:14:55.727 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T19:14:55.727 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T19:14:55.727 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T19:14:55.727 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T19:14:55.727 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T19:14:55.728 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T19:14:55.728 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T19:14:55.728 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T19:14:55.728 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T19:14:55.728 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T19:14:55.729 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T19:14:55.729 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T19:14:55.729 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T19:14:55.729 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T19:14:55.729 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T19:14:55.729 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T19:14:55.729 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T19:14:55.729 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T19:14:55.730 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T19:14:55.730 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T19:14:55.730 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T19:14:55.730 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T19:14:55.730 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T19:14:55.730 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T19:14:55.730 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T19:14:55.731 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T19:14:55.731 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T19:14:55.731 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T19:14:55.731 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T19:14:55.731 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T19:14:55.731 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T19:14:55.731 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T19:14:55.732 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T19:14:55.732 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T19:14:55.732 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T19:14:55.732 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T19:14:55.732 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T19:14:55.732 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T19:14:55.732 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T19:14:55.733 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T19:14:55.733 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T19:14:55.733 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T19:14:55.733 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T19:14:55.733 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T19:14:55.734 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T19:14:55.734 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T19:14:55.734 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T19:14:55.734 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T19:14:55.734 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T19:14:55.735 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T19:14:55.735 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T19:14:55.735 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T19:14:55.735 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T19:14:55.735 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T19:14:55.735 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T19:14:55.735 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T19:14:55.735 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T19:14:55.736 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T19:14:55.736 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T19:14:55.736 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T19:14:55.736 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T19:14:55.736 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T19:14:55.736 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T19:14:55.736 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T19:14:55.737 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T19:14:55.737 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T19:14:55.737 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T19:14:55.737 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T19:14:55.737 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T19:14:55.737 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T19:14:55.737 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T19:14:55.738 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T19:14:55.738 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T19:14:55.738 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T19:14:55.738 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T19:14:55.738 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T19:14:55.738 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T19:14:55.738 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T19:14:55.738 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T19:14:55.739 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:14:55.739 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T19:14:58.126 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:14:58.127 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:03:29 ago on Fri 14 Jul 2023 07:11:29 PM UTC. 2023-07-14T19:15:00.819 INFO:teuthology.orchestra.run.smithi053.stdout:Installed products updated. 2023-07-14T19:15:01.050 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:15:01.050 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2023-07-14T19:15:01.050 INFO:teuthology.orchestra.run.smithi053.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T19:15:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.051 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-07-14T19:15:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T19:15:01.051 INFO:teuthology.orchestra.run.smithi053.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T19:15:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T19:15:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.053 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.054 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.055 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.056 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.056 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.056 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.056 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:15:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T19:15:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T19:15:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T19:15:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T19:15:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: jq-1.6-3.el8.x86_64 2023-07-14T19:15:01.057 INFO:teuthology.orchestra.run.smithi053.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T19:15:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T19:15:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T19:15:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T19:15:01.058 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:15:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T19:15:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T19:15:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:15:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T19:15:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.059 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T19:15:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T19:15:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T19:15:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T19:15:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T19:15:01.060 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T19:15:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T19:15:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T19:15:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T19:15:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T19:15:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T19:15:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T19:15:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T19:15:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T19:15:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T19:15:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:15:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:15:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:15:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T19:15:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T19:15:01.062 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T19:15:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T19:15:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T19:15:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T19:15:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T19:15:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T19:15:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T19:15:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T19:15:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T19:15:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T19:15:01.064 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T19:15:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T19:15:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T19:15:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T19:15:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T19:15:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T19:15:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T19:15:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T19:15:01.065 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T19:15:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T19:15:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T19:15:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T19:15:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T19:15:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T19:15:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T19:15:01.066 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T19:15:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T19:15:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T19:15:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T19:15:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T19:15:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T19:15:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T19:15:01.067 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T19:15:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T19:15:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T19:15:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T19:15:01.068 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T19:15:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T19:15:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T19:15:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T19:15:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T19:15:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T19:15:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T19:15:01.069 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T19:15:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T19:15:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T19:15:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T19:15:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.070 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:15:01.071 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T19:15:01.071 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T19:15:01.071 INFO:teuthology.orchestra.run.smithi053.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T19:15:01.071 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T19:15:01.071 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T19:15:01.072 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T19:15:01.072 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:15:01.072 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-07-14T19:15:01.246 DEBUG:teuthology.parallel:result is None 2023-07-14T19:15:01.248 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:15:06.887 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T19:15:06.917 INFO:teuthology.orchestra.run.smithi053.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T19:15:06.918 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T19:15:06.918 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T19:15:06.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:15:12.544 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T19:15:12.573 INFO:teuthology.orchestra.run.smithi070.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T19:15:12.574 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T19:15:12.574 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T19:15:12.575 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:15:17.080 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T19:15:17.114 INFO:teuthology.orchestra.run.smithi104.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T19:15:17.114 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T19:15:17.115 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T19:15:17.115 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-14T19:15:17.116 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:15:17.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T19:15:17.166 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:15:17.166 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T19:15:17.214 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:15:17.214 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T19:15:17.261 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-14T19:15:17.262 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:15:17.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T19:15:17.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T19:15:17.366 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:15:17.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T19:15:17.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T19:15:17.468 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:15:17.468 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T19:15:17.499 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T19:15:17.568 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-14T19:15:17.568 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:15:17.568 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T19:15:17.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T19:15:17.670 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:15:17.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T19:15:17.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T19:15:17.769 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:15:17.770 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T19:15:17.800 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T19:15:17.869 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-14T19:15:17.870 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:15:17.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T19:15:17.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T19:15:17.975 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:15:17.975 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T19:15:18.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T19:15:18.073 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:15:18.073 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T19:15:18.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T19:15:18.172 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-14T19:15:18.274 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'subvols': {'create': 2}} 2023-07-14T19:15:18.276 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:15:18.287 INFO:tasks.cephadm:Cluster fsid is c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:15:18.288 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-14T19:15:18.289 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.53', 'mon.b': '172.21.15.70', 'mon.c': '172.21.15.104'} 2023-07-14T19:15:18.290 INFO:tasks.cephadm:First mon is mon.a on smithi053 2023-07-14T19:15:18.290 INFO:tasks.cephadm:First mgr is x 2023-07-14T19:15:18.291 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-14T19:15:18.291 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname $(hostname -s) 2023-07-14T19:15:18.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo hostname $(hostname -s) 2023-07-14T19:15:18.357 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2023-07-14T19:15:18.389 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-14T19:15:18.390 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:15:23.770 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71431/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-4940-g81c9387d', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-4940.g81c9387d'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-14 16:50:26.249583', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230714.134335', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2023-07-14T19:15:29.127 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20230714.134335, sha1 81c9387d2e7576da065747ec9c16d1c7d69016ec from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:15:29.128 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm 2023-07-14T19:15:29.128 DEBUG:teuthology.orchestra.run.smithi053:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T19:15:31.670 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 19:15 /home/ubuntu/cephtest/cephadm 2023-07-14T19:15:31.671 DEBUG:teuthology.orchestra.run.smithi070:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T19:15:34.367 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 19:15 /home/ubuntu/cephtest/cephadm 2023-07-14T19:15:34.367 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T19:15:38.095 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 19:15 /home/ubuntu/cephtest/cephadm 2023-07-14T19:15:38.096 DEBUG:teuthology.orchestra.run.smithi053:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T19:15:38.119 DEBUG:teuthology.orchestra.run.smithi070:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T19:15:38.140 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T19:15:38.175 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec on all hosts... 2023-07-14T19:15:38.175 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T19:15:38.178 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T19:15:38.181 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T19:15:39.386 INFO:teuthology.orchestra.run.smithi070.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:15:39.441 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:15:39.584 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:16:08.622 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-07-14T19:16:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:16:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T19:16:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: "repo_digests": [ 2023-07-14T19:16:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T19:16:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2023-07-14T19:16:08.624 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-07-14T19:16:36.673 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2023-07-14T19:16:36.673 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:16:36.673 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T19:16:36.674 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2023-07-14T19:16:36.674 INFO:teuthology.orchestra.run.smithi104.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T19:16:36.674 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2023-07-14T19:16:36.674 INFO:teuthology.orchestra.run.smithi104.stdout:} 2023-07-14T19:16:37.014 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2023-07-14T19:16:37.014 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:16:37.014 INFO:teuthology.orchestra.run.smithi070.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T19:16:37.014 INFO:teuthology.orchestra.run.smithi070.stdout: "repo_digests": [ 2023-07-14T19:16:37.015 INFO:teuthology.orchestra.run.smithi070.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T19:16:37.015 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2023-07-14T19:16:37.015 INFO:teuthology.orchestra.run.smithi070.stdout:} 2023-07-14T19:16:37.033 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph 2023-07-14T19:16:37.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph 2023-07-14T19:16:37.101 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2023-07-14T19:16:37.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /etc/ceph 2023-07-14T19:16:37.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /etc/ceph 2023-07-14T19:16:37.193 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2023-07-14T19:16:37.219 INFO:tasks.cephadm:Writing seed config... 2023-07-14T19:16:37.221 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-14T19:16:37.221 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-14T19:16:37.221 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-14T19:16:37.222 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-14T19:16:37.222 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-14T19:16:37.222 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-14T19:16:37.222 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-14T19:16:37.223 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-14T19:16:37.223 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-14T19:16:37.223 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-14T19:16:37.223 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-14T19:16:37.223 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-14T19:16:37.224 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-14T19:16:37.224 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-14T19:16:37.224 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-14T19:16:37.224 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-14T19:16:37.224 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-14T19:16:37.225 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-14T19:16:37.225 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-14T19:16:37.225 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-14T19:16:37.225 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-14T19:16:37.225 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-07-14T19:16:37.226 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-14T19:16:37.226 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-14T19:16:37.226 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-14T19:16:37.226 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-14T19:16:37.226 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-14T19:16:37.227 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-14T19:16:37.227 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-07-14T19:16:37.227 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-07-14T19:16:37.227 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-07-14T19:16:37.227 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-14T19:16:37.227 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-14T19:16:37.228 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-07-14T19:16:37.228 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-07-14T19:16:37.228 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-14T19:16:37.228 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-14T19:16:37.229 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-07-14T19:16:37.229 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-14T19:16:37.229 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-14T19:16:37.229 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-14T19:16:37.230 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-14T19:16:37.230 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-14T19:16:37.230 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-14T19:16:37.230 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-14T19:16:37.231 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-07-14T19:16:37.231 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:16:37.232 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-14T19:16:37.251 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c5147628-227a-11ee-9b33-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-14T19:16:37.252 DEBUG:teuthology.orchestra.run.smithi053:mon.a> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.a.service 2023-07-14T19:16:37.294 DEBUG:teuthology.orchestra.run.smithi053:mgr.x> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.x.service 2023-07-14T19:16:37.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:16:37.326 INFO:tasks.cephadm:Bootstrapping... 2023-07-14T19:16:37.327 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec -v bootstrap --fsid c5147628-227a-11ee-9b33-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.53 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:16:37.355 INFO:journalctl@ceph.mgr.x.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:16:37.499 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:16:37.499 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec', '-v', 'bootstrap', '--fsid', 'c5147628-227a-11ee-9b33-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.53', '--skip-admin-label'] 2023-07-14T19:16:37.643 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 4.1.1 2023-07-14T19:16:37.644 INFO:teuthology.orchestra.run.smithi053.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-14T19:16:37.644 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying podman|docker is present... 2023-07-14T19:16:37.788 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 4.1.1 2023-07-14T19:16:37.788 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying lvm2 is present... 2023-07-14T19:16:37.788 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying time synchronization is in place... 2023-07-14T19:16:37.795 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T19:16:37.795 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T19:16:37.801 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T19:16:37.801 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2023-07-14T19:16:37.807 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout enabled 2023-07-14T19:16:37.814 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout active 2023-07-14T19:16:37.815 INFO:teuthology.orchestra.run.smithi053.stdout:Unit chronyd.service is enabled and running 2023-07-14T19:16:37.815 INFO:teuthology.orchestra.run.smithi053.stdout:Repeating the final host check... 2023-07-14T19:16:37.960 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 4.1.1 2023-07-14T19:16:37.960 INFO:teuthology.orchestra.run.smithi053.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-14T19:16:37.960 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl is present 2023-07-14T19:16:37.960 INFO:teuthology.orchestra.run.smithi053.stdout:lvcreate is present 2023-07-14T19:16:37.967 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T19:16:37.967 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T19:16:37.972 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T19:16:37.973 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2023-07-14T19:16:37.979 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout enabled 2023-07-14T19:16:37.985 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout active 2023-07-14T19:16:37.985 INFO:teuthology.orchestra.run.smithi053.stdout:Unit chronyd.service is enabled and running 2023-07-14T19:16:37.985 INFO:teuthology.orchestra.run.smithi053.stdout:Host looks OK 2023-07-14T19:16:37.985 INFO:teuthology.orchestra.run.smithi053.stdout:Cluster fsid: c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:16:37.986 INFO:teuthology.orchestra.run.smithi053.stdout:Acquiring lock 140138305907176 on /run/cephadm/c5147628-227a-11ee-9b33-001a4aab830c.lock 2023-07-14T19:16:37.986 INFO:teuthology.orchestra.run.smithi053.stdout:Lock 140138305907176 acquired on /run/cephadm/c5147628-227a-11ee-9b33-001a4aab830c.lock 2023-07-14T19:16:37.986 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying IP 172.21.15.53 port 3300 ... 2023-07-14T19:16:37.987 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying IP 172.21.15.53 port 6789 ... 2023-07-14T19:16:37.987 INFO:teuthology.orchestra.run.smithi053.stdout:Base mon IP(s) is [172.21.15.53:3300, 172.21.15.53:6789], mon addrv is [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2023-07-14T19:16:37.990 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-07-14T19:16:37.990 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.53 metric 100 2023-07-14T19:16:37.993 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-14T19:16:37.993 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-14T19:16:37.993 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2023-07-14T19:16:37.996 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-14T19:16:37.996 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-14T19:16:37.996 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T19:16:37.996 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-14T19:16:37.996 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:3718/64 scope link 2023-07-14T19:16:37.997 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T19:16:37.997 INFO:teuthology.orchestra.run.smithi053.stdout:Mon IP `172.21.15.53` is in CIDR network `172.21.0.0/20` 2023-07-14T19:16:37.997 INFO:teuthology.orchestra.run.smithi053.stdout:Mon IP `172.21.15.53` is in CIDR network `172.21.0.0/20` 2023-07-14T19:16:37.997 INFO:teuthology.orchestra.run.smithi053.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-14T19:16:37.998 INFO:teuthology.orchestra.run.smithi053.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-14T19:16:37.999 INFO:teuthology.orchestra.run.smithi053.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:16:40.455 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T19:16:40.456 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:16:40.456 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Getting image source signatures 2023-07-14T19:16:40.456 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Copying blob sha256:0e454b422ba90e2ad13e636c40610401086a3576fe973b0460943839fe5f2508 2023-07-14T19:16:40.457 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-14T19:16:40.457 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Copying config sha256:219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T19:16:40.457 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-14T19:16:40.458 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Storing signatures 2023-07-14T19:16:42.012 INFO:teuthology.orchestra.run.smithi053.stdout:ceph: stdout ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T19:16:42.012 INFO:teuthology.orchestra.run.smithi053.stdout:Ceph version: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T19:16:42.013 INFO:teuthology.orchestra.run.smithi053.stdout:Extracting ceph user uid/gid from container image... 2023-07-14T19:16:42.860 INFO:teuthology.orchestra.run.smithi053.stdout:stat: stdout 167 167 2023-07-14T19:16:42.861 INFO:teuthology.orchestra.run.smithi053.stdout:Creating initial keys... 2023-07-14T19:16:43.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph-authtool: stdout AQAbn7Fks/IMERAAuEI8ZUwyZ7mYhToeKg8E2Q== 2023-07-14T19:16:44.461 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph-authtool: stdout AQAcn7Fkb1nGBhAAcI7aCOHS6ihwdsI/Dwmusg== 2023-07-14T19:16:45.210 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph-authtool: stdout AQAcn7FkQi/xMxAA7ddi4ERR2a95pYAtBVyo0Q== 2023-07-14T19:16:45.210 INFO:teuthology.orchestra.run.smithi053.stdout:Creating initial monmap... 2023-07-14T19:16:45.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T19:16:45.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-14T19:16:45.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:16:45.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T19:16:45.968 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool for a [v2:172.21.15.53:3300,v1:172.21.15.53:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T19:16:45.968 INFO:teuthology.orchestra.run.smithi053.stdout:setting min_mon_release = pacific 2023-07-14T19:16:45.968 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: set fsid to c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:16:45.968 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T19:16:45.968 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:16:45.968 INFO:teuthology.orchestra.run.smithi053.stdout:Creating mon... 2023-07-14T19:16:46.618 INFO:teuthology.orchestra.run.smithi053.stdout:create mon.a on 2023-07-14T19:16:46.761 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-14T19:16:46.854 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-14T19:16:46.982 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c5147628-227a-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-c5147628-227a-11ee-9b33-001a4aab830c.target. 2023-07-14T19:16:46.983 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c5147628-227a-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-c5147628-227a-11ee-9b33-001a4aab830c.target. 2023-07-14T19:16:47.162 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.a 2023-07-14T19:16:47.162 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.a.service: Unit ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.a.service not loaded. 2023-07-14T19:16:47.253 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c5147628-227a-11ee-9b33-001a4aab830c.target.wants/ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-c5147628-227a-11ee-9b33-001a4aab830c@.service. 2023-07-14T19:16:47.561 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:47 smithi053 systemd[1]: Starting Ceph mon.a for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:16:47.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:47 smithi053 podman[96010]: 2023-07-14T19:16:47.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:47 smithi053 bash[96010]: b4d0ac9a2aa04c745e350117b9e6c2388d445a1bf63d412097b9d3dcdd653eab 2023-07-14T19:16:47.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:47 smithi053 ceph-mon[96050]: mkfs c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:16:47.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:47 smithi053 ceph-mon[96050]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T19:16:47.862 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:16:47.863 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2023-07-14T19:16:47.870 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:16:47.870 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2023-07-14T19:16:47.870 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2023-07-14T19:16:47.870 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T19:16:47.871 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mon to start... 2023-07-14T19:16:47.871 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mon... 2023-07-14T19:16:48.103 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:47 smithi053 systemd[1]: Started Ceph mon.a for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:16:48.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout cluster: 2023-07-14T19:16:48.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout id: c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:16:48.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-14T19:16:48.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:16:48.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout services: 2023-07-14T19:16:48.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.686279s) 2023-07-14T19:16:48.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-14T19:16:48.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-14T19:16:48.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:16:48.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout data: 2023-07-14T19:16:48.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-14T19:16:48.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-14T19:16:48.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-14T19:16:48.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout pgs: 2023-07-14T19:16:48.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:16:48.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.341+0000 7fbe1ad0b700 1 Processor -- start 2023-07-14T19:16:48.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.341+0000 7fbe1ad0b700 1 -- start start 2023-07-14T19:16:48.824 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.341+0000 7fbe1ad0b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe141059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:48.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.341+0000 7fbe1ad0b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe14105ee0 con 0x7fbe14105590 2023-07-14T19:16:48.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.342+0000 7fbe18aa7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe141059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:48.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.342+0000 7fbe18aa7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe141059a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48872/0 (socket says 172.21.15.53:48872) 2023-07-14T19:16:48.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.342+0000 7fbe18aa7700 1 -- 172.21.15.53:0/3037108048 learned_addr learned my addr 172.21.15.53:0/3037108048 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:48.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.342+0000 7fbe18aa7700 1 -- 172.21.15.53:0/3037108048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe14106020 con 0x7fbe14105590 2023-07-14T19:16:48.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.342+0000 7fbe18aa7700 1 --2- 172.21.15.53:0/3037108048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe141059a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbe080098a0 tx=0x7fbe08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48450361b4af2f69 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:48.825 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.342+0000 7fbe137fe700 1 -- 172.21.15.53:0/3037108048 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe08004160 con 0x7fbe14105590 2023-07-14T19:16:48.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.342+0000 7fbe137fe700 1 -- 172.21.15.53:0/3037108048 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe080042c0 con 0x7fbe14105590 2023-07-14T19:16:48.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.342+0000 7fbe137fe700 1 -- 172.21.15.53:0/3037108048 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe080045a0 con 0x7fbe14105590 2023-07-14T19:16:48.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.343+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/3037108048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 msgr2=0x7fbe141059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:48.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.343+0000 7fbe1ad0b700 1 --2- 172.21.15.53:0/3037108048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe141059a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbe080098a0 tx=0x7fbe08009bb0 comp rx=0 tx=0).stop 2023-07-14T19:16:48.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.343+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/3037108048 shutdown_connections 2023-07-14T19:16:48.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.343+0000 7fbe1ad0b700 1 --2- 172.21.15.53:0/3037108048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe141059a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:48.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.343+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/3037108048 >> 172.21.15.53:0/3037108048 conn(0x7fbe14100c20 msgr2=0x7fbe14103030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:48.826 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.343+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/3037108048 shutdown_connections 2023-07-14T19:16:48.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.343+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/3037108048 wait complete. 2023-07-14T19:16:48.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe1ad0b700 1 Processor -- start 2023-07-14T19:16:48.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe1ad0b700 1 -- start start 2023-07-14T19:16:48.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe1ad0b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe14198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:48.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe1ad0b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe14198fe0 con 0x7fbe14105590 2023-07-14T19:16:48.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe18aa7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe14198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:48.827 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe18aa7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe14198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48874/0 (socket says 172.21.15.53:48874) 2023-07-14T19:16:48.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe18aa7700 1 -- 172.21.15.53:0/1848835647 learned_addr learned my addr 172.21.15.53:0/1848835647 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:48.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe18aa7700 1 -- 172.21.15.53:0/1848835647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe08009550 con 0x7fbe14105590 2023-07-14T19:16:48.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe18aa7700 1 --2- 172.21.15.53:0/1848835647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe14198aa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbe141062c0 tx=0x7fbe08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:48.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.344+0000 7fbe11ffb700 1 -- 172.21.15.53:0/1848835647 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe08011630 con 0x7fbe14105590 2023-07-14T19:16:48.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.345+0000 7fbe11ffb700 1 -- 172.21.15.53:0/1848835647 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe08011790 con 0x7fbe14105590 2023-07-14T19:16:48.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.345+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/1848835647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe141991e0 con 0x7fbe14105590 2023-07-14T19:16:48.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.345+0000 7fbe11ffb700 1 -- 172.21.15.53:0/1848835647 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe08011a30 con 0x7fbe14105590 2023-07-14T19:16:48.828 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.345+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/1848835647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe14199600 con 0x7fbe14105590 2023-07-14T19:16:48.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.345+0000 7fbe11ffb700 1 -- 172.21.15.53:0/1848835647 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbe08022070 con 0x7fbe14105590 2023-07-14T19:16:48.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.345+0000 7fbe11ffb700 1 -- 172.21.15.53:0/1848835647 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbe0801c450 con 0x7fbe14105590 2023-07-14T19:16:48.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.346+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/1848835647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe141922e0 con 0x7fbe14105590 2023-07-14T19:16:48.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.348+0000 7fbe11ffb700 1 -- 172.21.15.53:0/1848835647 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fbe0801b450 con 0x7fbe14105590 2023-07-14T19:16:48.829 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.479+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/1848835647 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fbe1402c8e0 con 0x7fbe14105590 2023-07-14T19:16:48.830 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.479+0000 7fbe11ffb700 1 -- 172.21.15.53:0/1848835647 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fbe0801b5f0 con 0x7fbe14105590 2023-07-14T19:16:48.830 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.480+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/1848835647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 msgr2=0x7fbe14198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:48.830 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.480+0000 7fbe1ad0b700 1 --2- 172.21.15.53:0/1848835647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe14198aa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbe141062c0 tx=0x7fbe08004750 comp rx=0 tx=0).stop 2023-07-14T19:16:48.830 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.480+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/1848835647 shutdown_connections 2023-07-14T19:16:48.830 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.480+0000 7fbe1ad0b700 1 --2- 172.21.15.53:0/1848835647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe14105590 0x7fbe14198aa0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:48.830 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.480+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/1848835647 >> 172.21.15.53:0/1848835647 conn(0x7fbe14100c20 msgr2=0x7fbe1418cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:48.831 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.480+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/1848835647 shutdown_connections 2023-07-14T19:16:48.831 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:48.480+0000 7fbe1ad0b700 1 -- 172.21.15.53:0/1848835647 wait complete. 2023-07-14T19:16:48.831 INFO:teuthology.orchestra.run.smithi053.stdout:mon is available 2023-07-14T19:16:48.831 INFO:teuthology.orchestra.run.smithi053.stdout:Assimilating anything we can from ceph.conf... 2023-07-14T19:16:49.053 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:48 smithi053 ceph-mon[96050]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T19:16:49.054 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:48 smithi053 ceph-mon[96050]: monmap e1: 1 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} removed_ranks: {} 2023-07-14T19:16:49.054 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:48 smithi053 ceph-mon[96050]: fsmap 2023-07-14T19:16:49.054 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:48 smithi053 ceph-mon[96050]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T19:16:49.054 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:48 smithi053 ceph-mon[96050]: mgrmap e1: no daemons active 2023-07-14T19:16:49.054 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:48 smithi053 ceph-mon[96050]: from='client.? 172.21.15.53:0/1848835647' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-14T19:16:49.873 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:16:49.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T19:16:49.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout fsid = c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:16:49.874 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2023-07-14T19:16:49.876 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T19:16:49.876 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T19:16:49.876 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T19:16:49.877 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T19:16:49.877 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:16:49.877 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T19:16:49.878 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T19:16:49.878 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:16:49.878 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T19:16:49.879 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T19:16:49.879 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T19:16:49.879 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T19:16:49.880 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T19:16:49.880 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T19:16:49.880 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.330+0000 7fb054250700 1 Processor -- start 2023-07-14T19:16:49.880 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.331+0000 7fb054250700 1 -- start start 2023-07-14T19:16:49.881 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.331+0000 7fb054250700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c104820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:49.881 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.331+0000 7fb054250700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c104d60 con 0x7fb04c104410 2023-07-14T19:16:49.881 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.331+0000 7fb051fec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c104820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:49.881 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.331+0000 7fb051fec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c104820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48876/0 (socket says 172.21.15.53:48876) 2023-07-14T19:16:49.881 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.331+0000 7fb051fec700 1 -- 172.21.15.53:0/155534331 learned_addr learned my addr 172.21.15.53:0/155534331 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:49.882 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.332+0000 7fb051fec700 1 -- 172.21.15.53:0/155534331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c104ea0 con 0x7fb04c104410 2023-07-14T19:16:49.882 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.332+0000 7fb051fec700 1 --2- 172.21.15.53:0/155534331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c104820 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0098a0 tx=0x7fb03c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0809d0775a4f3d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:49.882 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.332+0000 7fb050fea700 1 -- 172.21.15.53:0/155534331 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb03c004160 con 0x7fb04c104410 2023-07-14T19:16:49.883 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.332+0000 7fb050fea700 1 -- 172.21.15.53:0/155534331 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c0042c0 con 0x7fb04c104410 2023-07-14T19:16:49.883 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.332+0000 7fb050fea700 1 -- 172.21.15.53:0/155534331 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb03c0045a0 con 0x7fb04c104410 2023-07-14T19:16:49.883 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.332+0000 7fb054250700 1 -- 172.21.15.53:0/155534331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 msgr2=0x7fb04c104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:49.883 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.332+0000 7fb054250700 1 --2- 172.21.15.53:0/155534331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c104820 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0098a0 tx=0x7fb03c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:16:49.884 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.333+0000 7fb054250700 1 -- 172.21.15.53:0/155534331 shutdown_connections 2023-07-14T19:16:49.884 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.333+0000 7fb054250700 1 --2- 172.21.15.53:0/155534331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c104820 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:49.884 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.333+0000 7fb054250700 1 -- 172.21.15.53:0/155534331 >> 172.21.15.53:0/155534331 conn(0x7fb04c0ffac0 msgr2=0x7fb04c101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:49.884 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.333+0000 7fb054250700 1 -- 172.21.15.53:0/155534331 shutdown_connections 2023-07-14T19:16:49.885 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.333+0000 7fb054250700 1 -- 172.21.15.53:0/155534331 wait complete. 2023-07-14T19:16:49.885 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.333+0000 7fb054250700 1 Processor -- start 2023-07-14T19:16:49.885 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.333+0000 7fb054250700 1 -- start start 2023-07-14T19:16:49.885 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.333+0000 7fb054250700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:49.885 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.333+0000 7fb054250700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c19b7f0 con 0x7fb04c104410 2023-07-14T19:16:49.886 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb051fec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:49.886 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb051fec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c079ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48878/0 (socket says 172.21.15.53:48878) 2023-07-14T19:16:49.886 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb051fec700 1 -- 172.21.15.53:0/1619228466 learned_addr learned my addr 172.21.15.53:0/1619228466 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:49.886 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb051fec700 1 -- 172.21.15.53:0/1619228466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c009550 con 0x7fb04c104410 2023-07-14T19:16:49.886 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb051fec700 1 --2- 172.21.15.53:0/1619228466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c079ab0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb04c105140 tx=0x7fb03c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:49.886 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb042ffd700 1 -- 172.21.15.53:0/1619228466 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb03c011630 con 0x7fb04c104410 2023-07-14T19:16:49.886 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb042ffd700 1 -- 172.21.15.53:0/1619228466 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03c011790 con 0x7fb04c104410 2023-07-14T19:16:49.887 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb054250700 1 -- 172.21.15.53:0/1619228466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c19b930 con 0x7fb04c104410 2023-07-14T19:16:49.887 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb042ffd700 1 -- 172.21.15.53:0/1619228466 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb03c011a30 con 0x7fb04c104410 2023-07-14T19:16:49.887 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb054250700 1 -- 172.21.15.53:0/1619228466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c19bd50 con 0x7fb04c104410 2023-07-14T19:16:49.887 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.334+0000 7fb042ffd700 1 -- 172.21.15.53:0/1619228466 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb03c022070 con 0x7fb04c104410 2023-07-14T19:16:49.887 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.335+0000 7fb042ffd700 1 -- 172.21.15.53:0/1619228466 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb03c01c450 con 0x7fb04c104410 2023-07-14T19:16:49.887 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.336+0000 7fb054250700 1 -- 172.21.15.53:0/1619228466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04c1911d0 con 0x7fb04c104410 2023-07-14T19:16:49.887 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.338+0000 7fb042ffd700 1 -- 172.21.15.53:0/1619228466 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb03c01b450 con 0x7fb04c104410 2023-07-14T19:16:49.887 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.463+0000 7fb054250700 1 -- 172.21.15.53:0/1619228466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb04c02bf30 con 0x7fb04c104410 2023-07-14T19:16:49.888 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.467+0000 7fb042ffd700 1 -- 172.21.15.53:0/1619228466 <== mon.0 v2:172.21.15.53:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb03c01b5f0 con 0x7fb04c104410 2023-07-14T19:16:49.888 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.467+0000 7fb042ffd700 1 -- 172.21.15.53:0/1619228466 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fb03c024630 con 0x7fb04c104410 2023-07-14T19:16:49.888 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.468+0000 7fb054250700 1 -- 172.21.15.53:0/1619228466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 msgr2=0x7fb04c079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:49.888 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.468+0000 7fb054250700 1 --2- 172.21.15.53:0/1619228466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c079ab0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb04c105140 tx=0x7fb03c004750 comp rx=0 tx=0).stop 2023-07-14T19:16:49.888 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.468+0000 7fb054250700 1 -- 172.21.15.53:0/1619228466 shutdown_connections 2023-07-14T19:16:49.888 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.468+0000 7fb054250700 1 --2- 172.21.15.53:0/1619228466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c104410 0x7fb04c079ab0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:49.889 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.468+0000 7fb054250700 1 -- 172.21.15.53:0/1619228466 >> 172.21.15.53:0/1619228466 conn(0x7fb04c0ffac0 msgr2=0x7fb04c18be70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:49.889 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.468+0000 7fb054250700 1 -- 172.21.15.53:0/1619228466 shutdown_connections 2023-07-14T19:16:49.889 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:49.468+0000 7fb054250700 1 -- 172.21.15.53:0/1619228466 wait complete. 2023-07-14T19:16:49.889 INFO:teuthology.orchestra.run.smithi053.stdout:Generating new minimal ceph.conf... 2023-07-14T19:16:50.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:49 smithi053 ceph-mon[96050]: from='client.? 172.21.15.53:0/1619228466' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T19:16:50.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:49 smithi053 ceph-mon[96050]: from='client.? 172.21.15.53:0/1619228466' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-14T19:16:50.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.330+0000 7fa2fda24700 1 Processor -- start 2023-07-14T19:16:50.816 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.330+0000 7fa2fda24700 1 -- start start 2023-07-14T19:16:50.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.330+0000 7fa2fda24700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8103c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:50.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.330+0000 7fa2fda24700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8104160 con 0x7fa2f8103810 2023-07-14T19:16:50.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.331+0000 7fa2f6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:50.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.331+0000 7fa2f6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8103c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48880/0 (socket says 172.21.15.53:48880) 2023-07-14T19:16:50.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.331+0000 7fa2f6ffd700 1 -- 172.21.15.53:0/2338261290 learned_addr learned my addr 172.21.15.53:0/2338261290 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:50.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.331+0000 7fa2f6ffd700 1 -- 172.21.15.53:0/2338261290 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f81042a0 con 0x7fa2f8103810 2023-07-14T19:16:50.817 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.331+0000 7fa2f6ffd700 1 --2- 172.21.15.53:0/2338261290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8103c20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0098a0 tx=0x7fa2ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e6f532463e2abd server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:50.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.331+0000 7fa2f5ffb700 1 -- 172.21.15.53:0/2338261290 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2ec004160 con 0x7fa2f8103810 2023-07-14T19:16:50.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.331+0000 7fa2f5ffb700 1 -- 172.21.15.53:0/2338261290 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2ec0042c0 con 0x7fa2f8103810 2023-07-14T19:16:50.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.331+0000 7fa2f5ffb700 1 -- 172.21.15.53:0/2338261290 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2ec011a00 con 0x7fa2f8103810 2023-07-14T19:16:50.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.332+0000 7fa2fda24700 1 -- 172.21.15.53:0/2338261290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 msgr2=0x7fa2f8103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:50.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.332+0000 7fa2fda24700 1 --2- 172.21.15.53:0/2338261290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8103c20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec0098a0 tx=0x7fa2ec009bb0 comp rx=0 tx=0).stop 2023-07-14T19:16:50.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.332+0000 7fa2fda24700 1 -- 172.21.15.53:0/2338261290 shutdown_connections 2023-07-14T19:16:50.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.332+0000 7fa2fda24700 1 --2- 172.21.15.53:0/2338261290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8103c20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:50.818 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.332+0000 7fa2fda24700 1 -- 172.21.15.53:0/2338261290 >> 172.21.15.53:0/2338261290 conn(0x7fa2f80ff210 msgr2=0x7fa2f8101620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:50.819 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.332+0000 7fa2fda24700 1 -- 172.21.15.53:0/2338261290 shutdown_connections 2023-07-14T19:16:50.819 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.332+0000 7fa2fda24700 1 -- 172.21.15.53:0/2338261290 wait complete. 2023-07-14T19:16:50.819 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.333+0000 7fa2fda24700 1 Processor -- start 2023-07-14T19:16:50.819 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.333+0000 7fa2fda24700 1 -- start start 2023-07-14T19:16:50.819 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.333+0000 7fa2fda24700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:50.819 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.333+0000 7fa2fda24700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f8199880 con 0x7fa2f8103810 2023-07-14T19:16:50.819 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.333+0000 7fa2f6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:50.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.333+0000 7fa2f6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48882/0 (socket says 172.21.15.53:48882) 2023-07-14T19:16:50.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.333+0000 7fa2f6ffd700 1 -- 172.21.15.53:0/1535389710 learned_addr learned my addr 172.21.15.53:0/1535389710 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:50.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.333+0000 7fa2f6ffd700 1 -- 172.21.15.53:0/1535389710 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec009550 con 0x7fa2f8103810 2023-07-14T19:16:50.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.333+0000 7fa2f6ffd700 1 --2- 172.21.15.53:0/1535389710 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8199340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009fb0 tx=0x7fa2ec011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:50.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.334+0000 7fa2fca22700 1 -- 172.21.15.53:0/1535389710 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2ec003930 con 0x7fa2f8103810 2023-07-14T19:16:50.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.334+0000 7fa2fca22700 1 -- 172.21.15.53:0/1535389710 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2ec003a90 con 0x7fa2f8103810 2023-07-14T19:16:50.820 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.334+0000 7fa2fda24700 1 -- 172.21.15.53:0/1535389710 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f8199a80 con 0x7fa2f8103810 2023-07-14T19:16:50.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.334+0000 7fa2fca22700 1 -- 172.21.15.53:0/1535389710 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2ec0195e0 con 0x7fa2f8103810 2023-07-14T19:16:50.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.334+0000 7fa2fda24700 1 -- 172.21.15.53:0/1535389710 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f8199ea0 con 0x7fa2f8103810 2023-07-14T19:16:50.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.334+0000 7fa2fca22700 1 -- 172.21.15.53:0/1535389710 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa2ec019bc0 con 0x7fa2f8103810 2023-07-14T19:16:50.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.334+0000 7fa2fca22700 1 -- 172.21.15.53:0/1535389710 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa2ec0243f0 con 0x7fa2f8103810 2023-07-14T19:16:50.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.335+0000 7fa2fda24700 1 -- 172.21.15.53:0/1535389710 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f81932f0 con 0x7fa2f8103810 2023-07-14T19:16:50.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.337+0000 7fa2fca22700 1 -- 172.21.15.53:0/1535389710 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa2ec003c00 con 0x7fa2f8103810 2023-07-14T19:16:50.821 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.464+0000 7fa2fda24700 1 -- 172.21.15.53:0/1535389710 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa2f8107550 con 0x7fa2f8103810 2023-07-14T19:16:50.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.464+0000 7fa2fca22700 1 -- 172.21.15.53:0/1535389710 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fa2ec01a9d0 con 0x7fa2f8103810 2023-07-14T19:16:50.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.465+0000 7fa2fda24700 1 -- 172.21.15.53:0/1535389710 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 msgr2=0x7fa2f8199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:50.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.465+0000 7fa2fda24700 1 --2- 172.21.15.53:0/1535389710 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8199340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009fb0 tx=0x7fa2ec011ad0 comp rx=0 tx=0).stop 2023-07-14T19:16:50.822 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.465+0000 7fa2fda24700 1 -- 172.21.15.53:0/1535389710 shutdown_connections 2023-07-14T19:16:50.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.465+0000 7fa2fda24700 1 --2- 172.21.15.53:0/1535389710 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f8103810 0x7fa2f8199340 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:50.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.465+0000 7fa2fda24700 1 -- 172.21.15.53:0/1535389710 >> 172.21.15.53:0/1535389710 conn(0x7fa2f80ff210 msgr2=0x7fa2f8100960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:50.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.465+0000 7fa2fda24700 1 -- 172.21.15.53:0/1535389710 shutdown_connections 2023-07-14T19:16:50.823 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:50.465+0000 7fa2fda24700 1 -- 172.21.15.53:0/1535389710 wait complete. 2023-07-14T19:16:50.823 INFO:teuthology.orchestra.run.smithi053.stdout:Restarting the monitor... 2023-07-14T19:16:51.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:50 smithi053 ceph-mon[96050]: from='client.? 172.21.15.53:0/1535389710' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:16:51.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:50 smithi053 systemd[1]: Stopping Ceph mon.a for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:16:51.423 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:51 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-a[96026]: 2023-07-14T19:16:51.148+0000 7f9203830700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:16:51.423 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:51 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-a[96026]: 2023-07-14T19:16:51.148+0000 7f9203830700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-14T19:16:51.692 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:51 smithi053 bash[96391]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-a 2023-07-14T19:16:51.978 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:51 smithi053 systemd[1]: ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.a.service: Succeeded. 2023-07-14T19:16:51.979 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:51 smithi053 systemd[1]: Stopped Ceph mon.a for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:16:51.979 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:51 smithi053 systemd[1]: Starting Ceph mon.a for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:16:52.258 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:51 smithi053 podman[96517]: 2023-07-14T19:16:52.259 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T19:16:52.259 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T19:16:52.259 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 bash[96517]: b0966646b52bfff52fb2f3e2753736e87467c09f91a514e5f0c03020f1e531d6 2023-07-14T19:16:52.259 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: pidfile_write: ignore empty --pid-file 2023-07-14T19:16:52.259 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: load: jerasure load: lrc 2023-07-14T19:16:52.259 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: RocksDB version: 7.9.2 2023-07-14T19:16:52.259 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Git sha 0 2023-07-14T19:16:52.260 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T19:16:52.260 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: DB SUMMARY 2023-07-14T19:16:52.260 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: DB Session ID: SJHC0NT79DYVAE4OPC7Y 2023-07-14T19:16:52.260 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: CURRENT file: CURRENT 2023-07-14T19:16:52.260 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T19:16:52.260 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-14T19:16:52.261 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-14T19:16:52.261 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99497 ; 2023-07-14T19:16:52.261 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.error_if_exists: 0 2023-07-14T19:16:52.261 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.create_if_missing: 0 2023-07-14T19:16:52.261 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.paranoid_checks: 1 2023-07-14T19:16:52.262 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T19:16:52.262 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T19:16:52.262 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T19:16:52.262 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.env: 0x55b17b35dc60 2023-07-14T19:16:52.262 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T19:16:52.262 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.info_log: 0x55b17be8f2a0 2023-07-14T19:16:52.262 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T19:16:52.263 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.statistics: (nil) 2023-07-14T19:16:52.263 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.use_fsync: 0 2023-07-14T19:16:52.263 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_log_file_size: 0 2023-07-14T19:16:52.263 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T19:16:52.263 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T19:16:52.264 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T19:16:52.264 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T19:16:52.264 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.allow_fallocate: 1 2023-07-14T19:16:52.264 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T19:16:52.264 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T19:16:52.265 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.use_direct_reads: 0 2023-07-14T19:16:52.265 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T19:16:52.265 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T19:16:52.265 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.db_log_dir: 2023-07-14T19:16:52.265 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.wal_dir: 2023-07-14T19:16:52.265 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T19:16:52.265 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T19:16:52.266 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T19:16:52.266 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T19:16:52.266 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T19:16:52.266 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T19:16:52.266 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T19:16:52.266 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T19:16:52.266 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.write_buffer_manager: 0x55b17bdb03c0 2023-07-14T19:16:52.267 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T19:16:52.267 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T19:16:52.267 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T19:16:52.268 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T19:16:52.268 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T19:16:52.268 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T19:16:52.268 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T19:16:52.269 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T19:16:52.269 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.unordered_write: 0 2023-07-14T19:16:52.269 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T19:16:52.270 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T19:16:52.270 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T19:16:52.271 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T19:16:52.271 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.row_cache: None 2023-07-14T19:16:52.271 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.wal_filter: None 2023-07-14T19:16:52.271 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T19:16:52.272 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T19:16:52.272 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.two_write_queues: 0 2023-07-14T19:16:52.272 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T19:16:52.272 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.wal_compression: 0 2023-07-14T19:16:52.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.atomic_flush: 0 2023-07-14T19:16:52.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T19:16:52.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T19:16:52.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T19:16:52.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.log_readahead_size: 0 2023-07-14T19:16:52.274 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T19:16:52.274 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T19:16:52.274 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T19:16:52.274 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T19:16:52.274 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T19:16:52.274 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T19:16:52.275 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T19:16:52.275 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_background_jobs: 2 2023-07-14T19:16:52.275 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_background_compactions: -1 2023-07-14T19:16:52.275 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_subcompactions: 1 2023-07-14T19:16:52.275 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T19:16:52.275 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T19:16:52.275 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T19:16:52.275 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T19:16:52.276 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T19:16:52.276 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T19:16:52.276 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T19:16:52.276 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T19:16:52.276 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_open_files: -1 2023-07-14T19:16:52.276 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T19:16:52.276 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T19:16:52.277 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T19:16:52.277 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T19:16:52.277 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_background_flushes: -1 2023-07-14T19:16:52.277 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Compression algorithms supported: 2023-07-14T19:16:52.277 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T19:16:52.277 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: kZSTD supported: 0 2023-07-14T19:16:52.278 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: kXpressCompression supported: 0 2023-07-14T19:16:52.278 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T19:16:52.278 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: kZlibCompression supported: 1 2023-07-14T19:16:52.278 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: kSnappyCompression supported: 1 2023-07-14T19:16:52.278 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: kLZ4Compression supported: 1 2023-07-14T19:16:52.278 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: kBZip2Compression supported: 0 2023-07-14T19:16:52.278 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T19:16:52.278 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T19:16:52.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-14T19:16:52.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T19:16:52.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T19:16:52.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.merge_operator: 2023-07-14T19:16:52.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_filter: None 2023-07-14T19:16:52.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T19:16:52.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T19:16:52.280 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T19:16:52.280 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T19:16:52.280 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b17be8f3a0) 2023-07-14T19:16:52.280 INFO:journalctl@ceph.mon.a.smithi053.stdout: cache_index_and_filter_blocks: 1 2023-07-14T19:16:52.280 INFO:journalctl@ceph.mon.a.smithi053.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T19:16:52.280 INFO:journalctl@ceph.mon.a.smithi053.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T19:16:52.280 INFO:journalctl@ceph.mon.a.smithi053.stdout: pin_top_level_index_and_filter: 1 2023-07-14T19:16:52.281 INFO:journalctl@ceph.mon.a.smithi053.stdout: index_type: 0 2023-07-14T19:16:52.281 INFO:journalctl@ceph.mon.a.smithi053.stdout: data_block_index_type: 0 2023-07-14T19:16:52.281 INFO:journalctl@ceph.mon.a.smithi053.stdout: index_shortening: 1 2023-07-14T19:16:52.281 INFO:journalctl@ceph.mon.a.smithi053.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T19:16:52.281 INFO:journalctl@ceph.mon.a.smithi053.stdout: checksum: 4 2023-07-14T19:16:52.281 INFO:journalctl@ceph.mon.a.smithi053.stdout: no_block_cache: 0 2023-07-14T19:16:52.281 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache: 0x55b17be4f1f0 2023-07-14T19:16:52.281 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache_name: BinnedLRUCache 2023-07-14T19:16:52.282 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache_options: 2023-07-14T19:16:52.282 INFO:journalctl@ceph.mon.a.smithi053.stdout: capacity : 536870912 2023-07-14T19:16:52.282 INFO:journalctl@ceph.mon.a.smithi053.stdout: num_shard_bits : 4 2023-07-14T19:16:52.282 INFO:journalctl@ceph.mon.a.smithi053.stdout: strict_capacity_limit : 0 2023-07-14T19:16:52.282 INFO:journalctl@ceph.mon.a.smithi053.stdout: high_pri_pool_ratio: 0.000 2023-07-14T19:16:52.282 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache_compressed: (nil) 2023-07-14T19:16:52.282 INFO:journalctl@ceph.mon.a.smithi053.stdout: persistent_cache: (nil) 2023-07-14T19:16:52.283 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_size: 4096 2023-07-14T19:16:52.283 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_size_deviation: 10 2023-07-14T19:16:52.283 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_restart_interval: 16 2023-07-14T19:16:52.283 INFO:journalctl@ceph.mon.a.smithi053.stdout: index_block_restart_interval: 1 2023-07-14T19:16:52.283 INFO:journalctl@ceph.mon.a.smithi053.stdout: metadata_block_size: 4096 2023-07-14T19:16:52.283 INFO:journalctl@ceph.mon.a.smithi053.stdout: partition_filters: 0 2023-07-14T19:16:52.283 INFO:journalctl@ceph.mon.a.smithi053.stdout: use_delta_encoding: 1 2023-07-14T19:16:52.283 INFO:journalctl@ceph.mon.a.smithi053.stdout: filter_policy: bloomfilter 2023-07-14T19:16:52.284 INFO:journalctl@ceph.mon.a.smithi053.stdout: whole_key_filtering: 1 2023-07-14T19:16:52.284 INFO:journalctl@ceph.mon.a.smithi053.stdout: verify_compression: 0 2023-07-14T19:16:52.284 INFO:journalctl@ceph.mon.a.smithi053.stdout: read_amp_bytes_per_bit: 0 2023-07-14T19:16:52.284 INFO:journalctl@ceph.mon.a.smithi053.stdout: format_version: 5 2023-07-14T19:16:52.284 INFO:journalctl@ceph.mon.a.smithi053.stdout: enable_index_compression: 1 2023-07-14T19:16:52.284 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_align: 0 2023-07-14T19:16:52.285 INFO:journalctl@ceph.mon.a.smithi053.stdout: max_auto_readahead_size: 262144 2023-07-14T19:16:52.285 INFO:journalctl@ceph.mon.a.smithi053.stdout: prepopulate_block_cache: 0 2023-07-14T19:16:52.285 INFO:journalctl@ceph.mon.a.smithi053.stdout: initial_auto_readahead_size: 8192 2023-07-14T19:16:52.285 INFO:journalctl@ceph.mon.a.smithi053.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T19:16:52.285 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T19:16:52.285 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T19:16:52.285 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression: NoCompression 2023-07-14T19:16:52.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T19:16:52.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T19:16:52.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T19:16:52.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.num_levels: 7 2023-07-14T19:16:52.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T19:16:52.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T19:16:52.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T19:16:52.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T19:16:52.287 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T19:16:52.287 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T19:16:52.287 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T19:16:52.288 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:16:52.288 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T19:16:52.288 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T19:16:52.288 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:16:52.288 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:16:52.289 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T19:16:52.289 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T19:16:52.289 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T19:16:52.289 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T19:16:52.289 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:16:52.289 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:16:52.289 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T19:16:52.290 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T19:16:52.290 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:16:52.290 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T19:16:52.290 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T19:16:52.290 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T19:16:52.290 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T19:16:52.290 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T19:16:52.290 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T19:16:52.291 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T19:16:52.291 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T19:16:52.291 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T19:16:52.291 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T19:16:52.291 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T19:16:52.291 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T19:16:52.292 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T19:16:52.292 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T19:16:52.292 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T19:16:52.292 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T19:16:52.292 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T19:16:52.292 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T19:16:52.292 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T19:16:52.293 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T19:16:52.293 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T19:16:52.293 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T19:16:52.293 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T19:16:52.293 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T19:16:52.293 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T19:16:52.293 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T19:16:52.293 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T19:16:52.294 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T19:16:52.294 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T19:16:52.294 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T19:16:52.294 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T19:16:52.294 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T19:16:52.294 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.table_properties_collectors: 2023-07-14T19:16:52.295 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.inplace_update_support: 0 2023-07-14T19:16:52.295 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T19:16:52.295 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T19:16:52.295 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T19:16:52.295 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T19:16:52.295 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.bloom_locality: 0 2023-07-14T19:16:52.295 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.max_successive_merges: 0 2023-07-14T19:16:52.296 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T19:16:52.296 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T19:16:52.296 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T19:16:52.296 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T19:16:52.296 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.ttl: 2592000 2023-07-14T19:16:52.296 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T19:16:52.296 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T19:16:52.296 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T19:16:52.297 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.enable_blob_files: false 2023-07-14T19:16:52.297 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.min_blob_size: 0 2023-07-14T19:16:52.297 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T19:16:52.297 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T19:16:52.297 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T19:16:52.297 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T19:16:52.297 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T19:16:52.298 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T19:16:52.298 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T19:16:52.298 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T19:16:52.298 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-14T19:16:52.298 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-14T19:16:52.298 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: eeb22e0d-0b66-4b5f-a217-3960db6866b6 2023-07-14T19:16:52.298 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362212186110, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-14T19:16:52.298 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-14T19:16:52.299 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362212187168, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94660, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 318, "table_properties": {"data_size": 92651, "index_size": 291, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13858, "raw_average_key_size": 50, "raw_value_size": 85110, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 277, "num_filter_entries": 277, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689362212, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "eeb22e0d-0b66-4b5f-a217-3960db6866b6", "db_session_id": "SJHC0NT79DYVAE4OPC7Y", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-14T19:16:52.299 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362212187239, "job": 1, "event": "recovery_finished"} 2023-07-14T19:16:52.299 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-14T19:16:52.299 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T19:16:52.299 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b17bee8000 2023-07-14T19:16:52.299 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: DB pointer 0x55b17bed4000 2023-07-14T19:16:52.299 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T19:16:52.300 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T19:16:52.300 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** DB Stats ** 2023-07-14T19:16:52.300 INFO:journalctl@ceph.mon.a.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:16:52.300 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T19:16:52.300 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:16:52.300 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:16:52.301 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T19:16:52.301 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:16:52.301 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:16:52.301 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2023-07-14T19:16:52.301 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** Compaction Stats [default] ** 2023-07-14T19:16:52.301 INFO:journalctl@ceph.mon.a.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:16:52.302 INFO:journalctl@ceph.mon.a.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T19:16:52.302 INFO:journalctl@ceph.mon.a.smithi053.stdout: L0 2/0 94.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:16:52.302 INFO:journalctl@ceph.mon.a.smithi053.stdout: Sum 2/0 94.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:16:52.302 INFO:journalctl@ceph.mon.a.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:16:52.302 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2023-07-14T19:16:52.302 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** Compaction Stats [default] ** 2023-07-14T19:16:52.302 INFO:journalctl@ceph.mon.a.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:16:52.302 INFO:journalctl@ceph.mon.a.smithi053.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T19:16:52.303 INFO:journalctl@ceph.mon.a.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 135.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:16:52.303 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2023-07-14T19:16:52.303 INFO:journalctl@ceph.mon.a.smithi053.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T19:16:52.303 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2023-07-14T19:16:52.303 INFO:journalctl@ceph.mon.a.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:16:52.303 INFO:journalctl@ceph.mon.a.smithi053.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T19:16:52.303 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T19:16:52.304 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T19:16:52.304 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T19:16:52.304 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T19:16:52.304 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative compaction: 0.00 GB write, 28.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:16:52.304 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval compaction: 0.00 GB write, 28.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:16:52.305 INFO:journalctl@ceph.mon.a.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T19:16:52.305 INFO:journalctl@ceph.mon.a.smithi053.stdout: Block cache BinnedLRUCache@0x55b17be4f1f0#7 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-14T19:16:52.305 INFO:journalctl@ceph.mon.a.smithi053.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2023-07-14T19:16:52.305 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2023-07-14T19:16:52.305 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T19:16:52.305 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: starting mon.a rank 0 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:16:52.306 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mon.a@-1(???) e1 preinit fsid c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:16:52.306 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mon.a@-1(???).mds e1 new map 2023-07-14T19:16:52.306 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mon.a@-1(???).mds e1 print_map 2023-07-14T19:16:52.306 INFO:journalctl@ceph.mon.a.smithi053.stdout: e1 2023-07-14T19:16:52.306 INFO:journalctl@ceph.mon.a.smithi053.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:16:52.306 INFO:journalctl@ceph.mon.a.smithi053.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:16:52.306 INFO:journalctl@ceph.mon.a.smithi053.stdout: legacy client fscid: -1 2023-07-14T19:16:52.306 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2023-07-14T19:16:52.307 INFO:journalctl@ceph.mon.a.smithi053.stdout: No filesystems configured 2023-07-14T19:16:52.307 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T19:16:52.307 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:16:52.307 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:16:52.307 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:16:52.307 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-14T19:16:52.307 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T19:16:52.307 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: monmap e1: 1 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} removed_ranks: {} 2023-07-14T19:16:52.308 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: fsmap 2023-07-14T19:16:52.308 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T19:16:52.308 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 ceph-mon[96565]: mgrmap e1: no daemons active 2023-07-14T19:16:52.308 INFO:teuthology.orchestra.run.smithi053.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-14T19:16:52.513 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:52 smithi053 systemd[1]: Started Ceph mon.a for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:16:53.230 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.745+0000 7fa250a90700 1 Processor -- start 2023-07-14T19:16:53.234 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.745+0000 7fa250a90700 1 -- start start 2023-07-14T19:16:53.234 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.745+0000 7fa250a90700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c104820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.234 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.745+0000 7fa250a90700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c104d60 con 0x7fa24c104410 2023-07-14T19:16:53.234 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.746+0000 7fa24a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c104820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.234 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.746+0000 7fa24a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c104820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48884/0 (socket says 172.21.15.53:48884) 2023-07-14T19:16:53.234 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.746+0000 7fa24a59c700 1 -- 172.21.15.53:0/1004631922 learned_addr learned my addr 172.21.15.53:0/1004631922 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:53.235 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.746+0000 7fa24a59c700 1 -- 172.21.15.53:0/1004631922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c104ea0 con 0x7fa24c104410 2023-07-14T19:16:53.235 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.746+0000 7fa24a59c700 1 --2- 172.21.15.53:0/1004631922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c104820 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa23c0098a0 tx=0x7fa23c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab93e8a648d4030 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:53.235 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.746+0000 7fa24959a700 1 -- 172.21.15.53:0/1004631922 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa23c004160 con 0x7fa24c104410 2023-07-14T19:16:53.235 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.746+0000 7fa24959a700 1 -- 172.21.15.53:0/1004631922 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa23c0042c0 con 0x7fa24c104410 2023-07-14T19:16:53.235 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.746+0000 7fa24959a700 1 -- 172.21.15.53:0/1004631922 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa23c011a00 con 0x7fa24c104410 2023-07-14T19:16:53.235 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.747+0000 7fa250a90700 1 -- 172.21.15.53:0/1004631922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 msgr2=0x7fa24c104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.235 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.747+0000 7fa250a90700 1 --2- 172.21.15.53:0/1004631922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c104820 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa23c0098a0 tx=0x7fa23c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.235 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.747+0000 7fa250a90700 1 -- 172.21.15.53:0/1004631922 shutdown_connections 2023-07-14T19:16:53.236 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.747+0000 7fa250a90700 1 --2- 172.21.15.53:0/1004631922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c104820 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.236 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.747+0000 7fa250a90700 1 -- 172.21.15.53:0/1004631922 >> 172.21.15.53:0/1004631922 conn(0x7fa24c0ffac0 msgr2=0x7fa24c101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:53.236 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.747+0000 7fa250a90700 1 -- 172.21.15.53:0/1004631922 shutdown_connections 2023-07-14T19:16:53.236 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.747+0000 7fa250a90700 1 -- 172.21.15.53:0/1004631922 wait complete. 2023-07-14T19:16:53.236 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.748+0000 7fa250a90700 1 Processor -- start 2023-07-14T19:16:53.236 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.748+0000 7fa250a90700 1 -- start start 2023-07-14T19:16:53.236 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.748+0000 7fa250a90700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:53.237 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.748+0000 7fa250a90700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c1975c0 con 0x7fa24c104410 2023-07-14T19:16:53.237 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.748+0000 7fa24a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:53.238 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa24a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c197080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48886/0 (socket says 172.21.15.53:48886) 2023-07-14T19:16:53.238 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa24a59c700 1 -- 172.21.15.53:0/1141926539 learned_addr learned my addr 172.21.15.53:0/1141926539 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:53.238 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa24a59c700 1 -- 172.21.15.53:0/1141926539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c009550 con 0x7fa24c104410 2023-07-14T19:16:53.238 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa24a59c700 1 --2- 172.21.15.53:0/1141926539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c197080 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009fb0 tx=0x7fa23c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:53.238 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa23b7fe700 1 -- 172.21.15.53:0/1141926539 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa23c003930 con 0x7fa24c104410 2023-07-14T19:16:53.239 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa23b7fe700 1 -- 172.21.15.53:0/1141926539 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa23c003a90 con 0x7fa24c104410 2023-07-14T19:16:53.239 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa250a90700 1 -- 172.21.15.53:0/1141926539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24c1977c0 con 0x7fa24c104410 2023-07-14T19:16:53.239 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa23b7fe700 1 -- 172.21.15.53:0/1141926539 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa23c019610 con 0x7fa24c104410 2023-07-14T19:16:53.239 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa250a90700 1 -- 172.21.15.53:0/1141926539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24c197be0 con 0x7fa24c104410 2023-07-14T19:16:53.239 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa23b7fe700 1 -- 172.21.15.53:0/1141926539 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa23c019bf0 con 0x7fa24c104410 2023-07-14T19:16:53.239 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.749+0000 7fa23b7fe700 1 -- 172.21.15.53:0/1141926539 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa23c0243f0 con 0x7fa24c104410 2023-07-14T19:16:53.239 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.750+0000 7fa250a90700 1 -- 172.21.15.53:0/1141926539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24c190d40 con 0x7fa24c104410 2023-07-14T19:16:53.240 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.753+0000 7fa23b7fe700 1 -- 172.21.15.53:0/1141926539 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa23c003c00 con 0x7fa24c104410 2023-07-14T19:16:53.240 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.878+0000 7fa250a90700 1 -- 172.21.15.53:0/1141926539 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fa24c106440 con 0x7fa24c104410 2023-07-14T19:16:53.240 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.880+0000 7fa23b7fe700 1 -- 172.21.15.53:0/1141926539 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa23c01a9d0 con 0x7fa24c104410 2023-07-14T19:16:53.240 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.881+0000 7fa250a90700 1 -- 172.21.15.53:0/1141926539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 msgr2=0x7fa24c197080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:53.240 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.881+0000 7fa250a90700 1 --2- 172.21.15.53:0/1141926539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c197080 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009fb0 tx=0x7fa23c011ad0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.240 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.881+0000 7fa250a90700 1 -- 172.21.15.53:0/1141926539 shutdown_connections 2023-07-14T19:16:53.241 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.881+0000 7fa250a90700 1 --2- 172.21.15.53:0/1141926539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa24c104410 0x7fa24c197080 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:53.241 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.881+0000 7fa250a90700 1 -- 172.21.15.53:0/1141926539 >> 172.21.15.53:0/1141926539 conn(0x7fa24c0ffac0 msgr2=0x7fa24c18de80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:53.241 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.881+0000 7fa250a90700 1 -- 172.21.15.53:0/1141926539 shutdown_connections 2023-07-14T19:16:53.241 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:52.881+0000 7fa250a90700 1 -- 172.21.15.53:0/1141926539 wait complete. 2023-07-14T19:16:53.241 INFO:teuthology.orchestra.run.smithi053.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-14T19:16:53.258 INFO:teuthology.orchestra.run.smithi053.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:16:53.258 INFO:teuthology.orchestra.run.smithi053.stdout:Creating mgr... 2023-07-14T19:16:53.259 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying port 9283 ... 2023-07-14T19:16:53.259 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying port 8765 ... 2023-07-14T19:16:53.466 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T19:16:53.467 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.x.service: Unit ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.x.service not loaded. 2023-07-14T19:16:53.560 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c5147628-227a-11ee-9b33-001a4aab830c.target.wants/ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-c5147628-227a-11ee-9b33-001a4aab830c@.service. 2023-07-14T19:16:53.822 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:53 smithi053 systemd[1]: Starting Ceph mgr.x for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:16:54.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:53 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1141926539' entity='client.admin' 2023-07-14T19:16:54.155 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:53 smithi053 podman[96794]: 2023-07-14T19:16:54.156 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:54 smithi053 bash[96794]: f89b8bbe9363c3432fb4977de99eaf14736f20538b41d24547c7c0692937fada 2023-07-14T19:16:54.167 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:16:54.167 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2023-07-14T19:16:54.175 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:16:54.176 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2023-07-14T19:16:54.176 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2023-07-14T19:16:54.177 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T19:16:54.184 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:16:54.184 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2023-07-14T19:16:54.192 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:16:54.193 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2023-07-14T19:16:54.193 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2023-07-14T19:16:54.193 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-14T19:16:54.194 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr to start... 2023-07-14T19:16:54.194 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr... 2023-07-14T19:16:54.448 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:54 smithi053 systemd[1]: Started Ceph mgr.x for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:16:54.449 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:54 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:54.250+0000 7f3e31dd6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:16:54.449 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:54 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:54.346+0000 7f3e31dd6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:16:54.792 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:54 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:54.460+0000 7f3e31dd6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:16:54.793 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:54 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:54.575+0000 7f3e31dd6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:16:54.793 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:54 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:54.790+0000 7f3e31dd6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:16:55.056 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:54 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1722233515' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:16:55.251 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:16:55.251 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:16:55.251 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:16:55.251 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:16:55.252 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:16:55.252 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:16:55.252 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:16:55.252 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:55.252 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:16:55.252 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:16:55.253 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:16:55.253 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:16:55.253 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:16:55.253 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:16:55.253 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:16:55.253 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-07-14T19:16:55.253 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:16:55.254 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:16:55.254 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:16:55.254 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:16:55.254 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:55.254 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:16:55.254 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:16:55.254 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:16:55.255 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:16:55.255 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:16:55.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:16:55.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:16:55.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:16:55.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:55.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:16:55.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:16:55.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:16:55.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:16:55.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:16:55.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:16:55.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:16:55.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:16:55.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:16:55.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:55.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:16:55.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:16:55.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:16:55.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:16:55.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:55.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:16:55.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T19:16:55.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:16:55.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:16:55.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:16:55.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:16:55.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:16:55.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:16:55.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:16:55.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:55.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:16:55.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:16:55.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:16:47.795367+0000", 2023-07-14T19:16:55.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:16:55.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:55.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:16:55.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:16:55.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.761+0000 7f8365163700 1 Processor -- start 2023-07-14T19:16:55.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.761+0000 7f8365163700 1 -- start start 2023-07-14T19:16:55.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.761+0000 7f8365163700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836010a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.761+0000 7f8365163700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8360105ed0 con 0x7f8360073d40 2023-07-14T19:16:55.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.761+0000 7f835ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836010a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.761+0000 7f835ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836010a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48892/0 (socket says 172.21.15.53:48892) 2023-07-14T19:16:55.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.761+0000 7f835ed9d700 1 -- 172.21.15.53:0/2826861158 learned_addr learned my addr 172.21.15.53:0/2826861158 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:55.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.762+0000 7f835ed9d700 1 -- 172.21.15.53:0/2826861158 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836010aa20 con 0x7f8360073d40 2023-07-14T19:16:55.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.762+0000 7f835ed9d700 1 --2- 172.21.15.53:0/2826861158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836010a4e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f83500098a0 tx=0x7f8350009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c997fbb48bbf7ef3 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:55.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.762+0000 7f835dd9b700 1 -- 172.21.15.53:0/2826861158 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8350004160 con 0x7f8360073d40 2023-07-14T19:16:55.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.762+0000 7f835dd9b700 1 -- 172.21.15.53:0/2826861158 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83500042c0 con 0x7f8360073d40 2023-07-14T19:16:55.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.762+0000 7f835dd9b700 1 -- 172.21.15.53:0/2826861158 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8350011a00 con 0x7f8360073d40 2023-07-14T19:16:55.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.762+0000 7f8365163700 1 -- 172.21.15.53:0/2826861158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 msgr2=0x7f836010a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.762+0000 7f8365163700 1 --2- 172.21.15.53:0/2826861158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836010a4e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f83500098a0 tx=0x7f8350009bb0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.763+0000 7f8365163700 1 -- 172.21.15.53:0/2826861158 shutdown_connections 2023-07-14T19:16:55.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.763+0000 7f8365163700 1 --2- 172.21.15.53:0/2826861158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836010a4e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.763+0000 7f8365163700 1 -- 172.21.15.53:0/2826861158 >> 172.21.15.53:0/2826861158 conn(0x7f83600ffab0 msgr2=0x7f8360101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:55.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.763+0000 7f8365163700 1 -- 172.21.15.53:0/2826861158 shutdown_connections 2023-07-14T19:16:55.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.763+0000 7f8365163700 1 -- 172.21.15.53:0/2826861158 wait complete. 2023-07-14T19:16:55.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.763+0000 7f8365163700 1 Processor -- start 2023-07-14T19:16:55.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.764+0000 7f8365163700 1 -- start start 2023-07-14T19:16:55.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.764+0000 7f8365163700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836019b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:55.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.764+0000 7f8365163700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836019ba50 con 0x7f8360073d40 2023-07-14T19:16:55.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.764+0000 7f835ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836019b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:55.270 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.764+0000 7f835ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836019b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48894/0 (socket says 172.21.15.53:48894) 2023-07-14T19:16:55.270 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.764+0000 7f835ed9d700 1 -- 172.21.15.53:0/1722233515 learned_addr learned my addr 172.21.15.53:0/1722233515 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:55.270 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.764+0000 7f835ed9d700 1 -- 172.21.15.53:0/1722233515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8350009550 con 0x7f8360073d40 2023-07-14T19:16:55.270 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.764+0000 7f835ed9d700 1 --2- 172.21.15.53:0/1722233515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836019b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8350009fb0 tx=0x7f8350011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:55.270 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.765+0000 7f8347fff700 1 -- 172.21.15.53:0/1722233515 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f83500039c0 con 0x7f8360073d40 2023-07-14T19:16:55.270 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.765+0000 7f8347fff700 1 -- 172.21.15.53:0/1722233515 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8350003b20 con 0x7f8360073d40 2023-07-14T19:16:55.270 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.765+0000 7f8347fff700 1 -- 172.21.15.53:0/1722233515 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f835001a780 con 0x7f8360073d40 2023-07-14T19:16:55.271 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.765+0000 7f8365163700 1 -- 172.21.15.53:0/1722233515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836019bc50 con 0x7f8360073d40 2023-07-14T19:16:55.271 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.765+0000 7f8365163700 1 -- 172.21.15.53:0/1722233515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836019c070 con 0x7f8360073d40 2023-07-14T19:16:55.271 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.765+0000 7f8347fff700 1 -- 172.21.15.53:0/1722233515 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f835001abe0 con 0x7f8360073d40 2023-07-14T19:16:55.271 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.765+0000 7f8347fff700 1 -- 172.21.15.53:0/1722233515 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f83500193c0 con 0x7f8360073d40 2023-07-14T19:16:55.272 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.766+0000 7f8365163700 1 -- 172.21.15.53:0/1722233515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83601954f0 con 0x7f8360073d40 2023-07-14T19:16:55.272 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.768+0000 7f8347fff700 1 -- 172.21.15.53:0/1722233515 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f835002de40 con 0x7f8360073d40 2023-07-14T19:16:55.272 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.902+0000 7f8365163700 1 -- 172.21.15.53:0/1722233515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f836002c8e0 con 0x7f8360073d40 2023-07-14T19:16:55.272 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.902+0000 7f8347fff700 1 -- 172.21.15.53:0/1722233515 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f8350003c90 con 0x7f8360073d40 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.903+0000 7f8365163700 1 -- 172.21.15.53:0/1722233515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 msgr2=0x7f836019b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.903+0000 7f8365163700 1 --2- 172.21.15.53:0/1722233515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836019b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8350009fb0 tx=0x7f8350011ad0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.903+0000 7f8365163700 1 -- 172.21.15.53:0/1722233515 shutdown_connections 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.903+0000 7f8365163700 1 --2- 172.21.15.53:0/1722233515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8360073d40 0x7f836019b510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.903+0000 7f8365163700 1 -- 172.21.15.53:0/1722233515 >> 172.21.15.53:0/1722233515 conn(0x7f83600ffab0 msgr2=0x7f8360101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.903+0000 7f8365163700 1 -- 172.21.15.53:0/1722233515 shutdown_connections 2023-07-14T19:16:55.273 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:54.903+0000 7f8365163700 1 -- 172.21.15.53:0/1722233515 wait complete. 2023-07-14T19:16:55.274 INFO:teuthology.orchestra.run.smithi053.stdout:mgr not available, waiting (1/15)... 2023-07-14T19:16:55.317 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:55 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:55.204+0000 7f3e31dd6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:16:55.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:55 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:55.315+0000 7f3e31dd6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:16:55.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:55 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:55.467+0000 7f3e31dd6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:16:56.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:55 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:55.584+0000 7f3e31dd6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:16:56.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:56.099+0000 7f3e31dd6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:16:57.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:56.803+0000 7f3e31dd6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:16:57.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:56.995+0000 7f3e31dd6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:16:58.194 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:57 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:57.989+0000 7f3e31dd6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:16:58.195 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:58.101+0000 7f3e31dd6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:16:58.302 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:16:58.302 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:16:58.302 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:16:58.302 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:16:58.303 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:16:58.303 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:16:58.303 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:16:58.303 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:58.303 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:16:58.303 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:16:58.303 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:16:58.304 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:16:58.304 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:16:58.304 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:16:58.304 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:16:58.304 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-07-14T19:16:58.304 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:16:58.304 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:16:58.304 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:16:58.305 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:16:58.305 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:58.305 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:16:58.305 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:16:58.305 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:16:58.305 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:16:58.305 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:16:58.311 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:16:58.311 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:16:58.311 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:16:58.311 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:58.311 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:16:58.312 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:16:58.312 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:16:58.312 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:16:58.312 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:16:58.312 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:16:58.312 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:16:58.312 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:16:58.313 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:16:58.313 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:58.313 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:16:58.313 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:16:58.313 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:16:58.313 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:16:58.313 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:58.313 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:16:58.314 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T19:16:58.314 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:16:58.314 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:16:58.314 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:16:58.314 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:16:58.314 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:16:58.314 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:16:58.315 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:16:58.315 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:58.315 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:16:58.315 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:16:58.315 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:16:47.795367+0000", 2023-07-14T19:16:58.315 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:16:58.315 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:16:58.315 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:16:58.316 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:16:58.316 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.726+0000 7f7e9a473700 1 Processor -- start 2023-07-14T19:16:58.316 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.726+0000 7f7e9a473700 1 -- start start 2023-07-14T19:16:58.316 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.726+0000 7f7e9a473700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e94108470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:58.316 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.726+0000 7f7e9a473700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94073d80 con 0x7f7e94106090 2023-07-14T19:16:58.316 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.726+0000 7f7e93fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e94108470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:58.316 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.726+0000 7f7e93fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e94108470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48896/0 (socket says 172.21.15.53:48896) 2023-07-14T19:16:58.316 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.726+0000 7f7e93fff700 1 -- 172.21.15.53:0/2623437883 learned_addr learned my addr 172.21.15.53:0/2623437883 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:58.317 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.727+0000 7f7e93fff700 1 -- 172.21.15.53:0/2623437883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e94073ec0 con 0x7f7e94106090 2023-07-14T19:16:58.317 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.727+0000 7f7e93fff700 1 --2- 172.21.15.53:0/2623437883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e94108470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e840098a0 tx=0x7f7e84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d4d2348318fa0a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:58.317 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.727+0000 7f7e92ffd700 1 -- 172.21.15.53:0/2623437883 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e84004160 con 0x7f7e94106090 2023-07-14T19:16:58.317 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.727+0000 7f7e92ffd700 1 -- 172.21.15.53:0/2623437883 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e840042c0 con 0x7f7e94106090 2023-07-14T19:16:58.317 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.727+0000 7f7e92ffd700 1 -- 172.21.15.53:0/2623437883 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e84011a00 con 0x7f7e94106090 2023-07-14T19:16:58.317 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.728+0000 7f7e9a473700 1 -- 172.21.15.53:0/2623437883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 msgr2=0x7f7e94108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:58.317 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.728+0000 7f7e9a473700 1 --2- 172.21.15.53:0/2623437883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e94108470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e840098a0 tx=0x7f7e84009bb0 comp rx=0 tx=0).stop 2023-07-14T19:16:58.318 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.728+0000 7f7e9a473700 1 -- 172.21.15.53:0/2623437883 shutdown_connections 2023-07-14T19:16:58.318 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.728+0000 7f7e9a473700 1 --2- 172.21.15.53:0/2623437883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e94108470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:58.318 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.728+0000 7f7e9a473700 1 -- 172.21.15.53:0/2623437883 >> 172.21.15.53:0/2623437883 conn(0x7f7e940ffab0 msgr2=0x7f7e94101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:58.318 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.728+0000 7f7e9a473700 1 -- 172.21.15.53:0/2623437883 shutdown_connections 2023-07-14T19:16:58.318 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.728+0000 7f7e9a473700 1 -- 172.21.15.53:0/2623437883 wait complete. 2023-07-14T19:16:58.318 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.728+0000 7f7e9a473700 1 Processor -- start 2023-07-14T19:16:58.318 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.729+0000 7f7e9a473700 1 -- start start 2023-07-14T19:16:58.318 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.729+0000 7f7e9a473700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e941992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:16:58.319 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.729+0000 7f7e9a473700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94199810 con 0x7f7e94106090 2023-07-14T19:16:58.319 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.729+0000 7f7e93fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e941992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:16:58.319 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.729+0000 7f7e93fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e941992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48898/0 (socket says 172.21.15.53:48898) 2023-07-14T19:16:58.319 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.729+0000 7f7e93fff700 1 -- 172.21.15.53:0/3384552174 learned_addr learned my addr 172.21.15.53:0/3384552174 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:16:58.319 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.729+0000 7f7e93fff700 1 -- 172.21.15.53:0/3384552174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e84009550 con 0x7f7e94106090 2023-07-14T19:16:58.319 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.730+0000 7f7e93fff700 1 --2- 172.21.15.53:0/3384552174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e941992d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7e84006af0 tx=0x7f7e84011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:16:58.319 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.730+0000 7f7e917fa700 1 -- 172.21.15.53:0/3384552174 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e84003a40 con 0x7f7e94106090 2023-07-14T19:16:58.319 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.730+0000 7f7e917fa700 1 -- 172.21.15.53:0/3384552174 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e84003ba0 con 0x7f7e94106090 2023-07-14T19:16:58.320 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.730+0000 7f7e9a473700 1 -- 172.21.15.53:0/3384552174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e94199a10 con 0x7f7e94106090 2023-07-14T19:16:58.320 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.730+0000 7f7e917fa700 1 -- 172.21.15.53:0/3384552174 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e8401a700 con 0x7f7e94106090 2023-07-14T19:16:58.320 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.730+0000 7f7e9a473700 1 -- 172.21.15.53:0/3384552174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e94199e30 con 0x7f7e94106090 2023-07-14T19:16:58.320 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.730+0000 7f7e917fa700 1 -- 172.21.15.53:0/3384552174 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7e8401ac30 con 0x7f7e94106090 2023-07-14T19:16:58.320 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.730+0000 7f7e917fa700 1 -- 172.21.15.53:0/3384552174 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7e84019700 con 0x7f7e94106090 2023-07-14T19:16:58.320 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.731+0000 7f7e9a473700 1 -- 172.21.15.53:0/3384552174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e94061c70 con 0x7f7e94106090 2023-07-14T19:16:58.320 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.733+0000 7f7e917fa700 1 -- 172.21.15.53:0/3384552174 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f7e8402de40 con 0x7f7e94106090 2023-07-14T19:16:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.867+0000 7f7e9a473700 1 -- 172.21.15.53:0/3384552174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7e9419c870 con 0x7f7e94106090 2023-07-14T19:16:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.867+0000 7f7e917fa700 1 -- 172.21.15.53:0/3384552174 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f7e84024ad0 con 0x7f7e94106090 2023-07-14T19:16:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.868+0000 7f7e9a473700 1 -- 172.21.15.53:0/3384552174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 msgr2=0x7f7e941992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:16:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.868+0000 7f7e9a473700 1 --2- 172.21.15.53:0/3384552174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e941992d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7e84006af0 tx=0x7f7e84011ad0 comp rx=0 tx=0).stop 2023-07-14T19:16:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.869+0000 7f7e9a473700 1 -- 172.21.15.53:0/3384552174 shutdown_connections 2023-07-14T19:16:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.869+0000 7f7e9a473700 1 --2- 172.21.15.53:0/3384552174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e94106090 0x7f7e941992d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:16:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.869+0000 7f7e9a473700 1 -- 172.21.15.53:0/3384552174 >> 172.21.15.53:0/3384552174 conn(0x7f7e940ffab0 msgr2=0x7f7e94101170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:16:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.869+0000 7f7e9a473700 1 -- 172.21.15.53:0/3384552174 shutdown_connections 2023-07-14T19:16:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:16:57.869+0000 7f7e9a473700 1 -- 172.21.15.53:0/3384552174 wait complete. 2023-07-14T19:16:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:mgr not available, waiting (2/15)... 2023-07-14T19:16:58.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:16:58 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3384552174' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:16:58.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:58.323+0000 7f3e31dd6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:16:58.928 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:58.671+0000 7f3e31dd6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:16:59.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:58.926+0000 7f3e31dd6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:16:59.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:59 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:59.108+0000 7f3e31dd6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:16:59.822 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:16:59 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:16:59.436+0000 7f3e31dd6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:17:00.547 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:00.266+0000 7f3e31dd6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:17:00.547 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:00.429+0000 7f3e31dd6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:17:00.822 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:00.682+0000 7f3e31dd6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:17:01.114 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:00 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2065892403' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:17:01.114 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:00.824+0000 7f3e31dd6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:17:01.255 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:17:01.256 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:17:01.256 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:17:01.256 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:17:01.256 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:17:01.256 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:17:01.256 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:17:01.257 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:01.257 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:17:01.257 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:17:01.257 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:17:01.257 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:17:01.257 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:17:01.257 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:17:01.257 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:17:01.258 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-14T19:17:01.258 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:17:01.258 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:17:01.258 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:17:01.258 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:17:01.258 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:01.258 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:17:01.259 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:17:01.259 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:17:01.259 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:17:01.259 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:17:01.259 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:17:01.260 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:17:01.260 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:17:01.260 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:01.260 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:17:01.260 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:17:01.260 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:17:01.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:17:01.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:17:01.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:17:01.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:17:01.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:17:01.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:17:01.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:01.261 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:17:01.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:17:01.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:17:01.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:17:01.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:01.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:17:01.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T19:17:01.262 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:17:01.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:17:01.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:17:01.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:17:01.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:17:01.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:17:01.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:17:01.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:01.263 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:17:01.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:17:01.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:16:47.795367+0000", 2023-07-14T19:17:01.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:17:01.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:01.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:17:01.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:17:01.264 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.775+0000 7fa23b0cb700 1 Processor -- start 2023-07-14T19:17:01.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.775+0000 7fa23b0cb700 1 -- start start 2023-07-14T19:17:01.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.775+0000 7fa23b0cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234073d60 0x7fa2341080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:01.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.775+0000 7fa23b0cb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa234108600 con 0x7fa234073d60 2023-07-14T19:17:01.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.775+0000 7fa238e67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234073d60 0x7fa2341080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:01.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.775+0000 7fa238e67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234073d60 0x7fa2341080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48900/0 (socket says 172.21.15.53:48900) 2023-07-14T19:17:01.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.775+0000 7fa238e67700 1 -- 172.21.15.53:0/405609648 learned_addr learned my addr 172.21.15.53:0/405609648 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:01.265 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.775+0000 7fa238e67700 1 -- 172.21.15.53:0/405609648 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa234108740 con 0x7fa234073d60 2023-07-14T19:17:01.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.776+0000 7fa238e67700 1 --2- 172.21.15.53:0/405609648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234073d60 0x7fa2341080c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa2280098a0 tx=0x7fa228009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2844ac12c8e26c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:01.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.776+0000 7fa2337fe700 1 -- 172.21.15.53:0/405609648 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa228004160 con 0x7fa234073d60 2023-07-14T19:17:01.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.776+0000 7fa2337fe700 1 -- 172.21.15.53:0/405609648 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2280042c0 con 0x7fa234073d60 2023-07-14T19:17:01.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.776+0000 7fa2337fe700 1 -- 172.21.15.53:0/405609648 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa228011a00 con 0x7fa234073d60 2023-07-14T19:17:01.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.776+0000 7fa23b0cb700 1 -- 172.21.15.53:0/405609648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234073d60 msgr2=0x7fa2341080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:01.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.776+0000 7fa23b0cb700 1 --2- 172.21.15.53:0/405609648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234073d60 0x7fa2341080c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa2280098a0 tx=0x7fa228009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.776+0000 7fa23b0cb700 1 -- 172.21.15.53:0/405609648 shutdown_connections 2023-07-14T19:17:01.266 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.776+0000 7fa23b0cb700 1 --2- 172.21.15.53:0/405609648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234073d60 0x7fa2341080c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.776+0000 7fa23b0cb700 1 -- 172.21.15.53:0/405609648 >> 172.21.15.53:0/405609648 conn(0x7fa2340ffab0 msgr2=0x7fa234101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:01.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.777+0000 7fa23b0cb700 1 -- 172.21.15.53:0/405609648 shutdown_connections 2023-07-14T19:17:01.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.777+0000 7fa23b0cb700 1 -- 172.21.15.53:0/405609648 wait complete. 2023-07-14T19:17:01.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.777+0000 7fa23b0cb700 1 Processor -- start 2023-07-14T19:17:01.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.777+0000 7fa23b0cb700 1 -- start start 2023-07-14T19:17:01.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.777+0000 7fa23b0cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234199270 0x7fa234199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:01.267 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.777+0000 7fa23b0cb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa234199bc0 con 0x7fa234199270 2023-07-14T19:17:01.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.777+0000 7fa238e67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234199270 0x7fa234199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:01.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.778+0000 7fa238e67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234199270 0x7fa234199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48902/0 (socket says 172.21.15.53:48902) 2023-07-14T19:17:01.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.778+0000 7fa238e67700 1 -- 172.21.15.53:0/2065892403 learned_addr learned my addr 172.21.15.53:0/2065892403 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:01.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.778+0000 7fa238e67700 1 -- 172.21.15.53:0/2065892403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228009550 con 0x7fa234199270 2023-07-14T19:17:01.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.778+0000 7fa238e67700 1 --2- 172.21.15.53:0/2065892403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234199270 0x7fa234199680 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa228009fb0 tx=0x7fa228011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:01.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.778+0000 7fa231ffb700 1 -- 172.21.15.53:0/2065892403 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa228003930 con 0x7fa234199270 2023-07-14T19:17:01.268 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.778+0000 7fa231ffb700 1 -- 172.21.15.53:0/2065892403 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa228003a90 con 0x7fa234199270 2023-07-14T19:17:01.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.778+0000 7fa23b0cb700 1 -- 172.21.15.53:0/2065892403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa234199dc0 con 0x7fa234199270 2023-07-14T19:17:01.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.778+0000 7fa231ffb700 1 -- 172.21.15.53:0/2065892403 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa2280195e0 con 0x7fa234199270 2023-07-14T19:17:01.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.778+0000 7fa23b0cb700 1 -- 172.21.15.53:0/2065892403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23419ca20 con 0x7fa234199270 2023-07-14T19:17:01.269 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.779+0000 7fa231ffb700 1 -- 172.21.15.53:0/2065892403 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa228019b70 con 0x7fa234199270 2023-07-14T19:17:01.270 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.779+0000 7fa231ffb700 1 -- 172.21.15.53:0/2065892403 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa2280243f0 con 0x7fa234199270 2023-07-14T19:17:01.270 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.780+0000 7fa23b0cb700 1 -- 172.21.15.53:0/2065892403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa234061c70 con 0x7fa234199270 2023-07-14T19:17:01.271 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.782+0000 7fa231ffb700 1 -- 172.21.15.53:0/2065892403 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa22802de30 con 0x7fa234199270 2023-07-14T19:17:01.271 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.915+0000 7fa23b0cb700 1 -- 172.21.15.53:0/2065892403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa234073d60 con 0x7fa234199270 2023-07-14T19:17:01.271 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.915+0000 7fa231ffb700 1 -- 172.21.15.53:0/2065892403 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fa228003c00 con 0x7fa234199270 2023-07-14T19:17:01.271 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.917+0000 7fa23b0cb700 1 -- 172.21.15.53:0/2065892403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234199270 msgr2=0x7fa234199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:01.271 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.917+0000 7fa23b0cb700 1 --2- 172.21.15.53:0/2065892403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234199270 0x7fa234199680 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa228009fb0 tx=0x7fa228011ad0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.272 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.917+0000 7fa23b0cb700 1 -- 172.21.15.53:0/2065892403 shutdown_connections 2023-07-14T19:17:01.272 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.917+0000 7fa23b0cb700 1 --2- 172.21.15.53:0/2065892403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa234199270 0x7fa234199680 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:01.272 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.917+0000 7fa23b0cb700 1 -- 172.21.15.53:0/2065892403 >> 172.21.15.53:0/2065892403 conn(0x7fa2340ffab0 msgr2=0x7fa234101150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:01.272 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.917+0000 7fa23b0cb700 1 -- 172.21.15.53:0/2065892403 shutdown_connections 2023-07-14T19:17:01.273 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:00.917+0000 7fa23b0cb700 1 -- 172.21.15.53:0/2065892403 wait complete. 2023-07-14T19:17:01.273 INFO:teuthology.orchestra.run.smithi053.stdout:mgr not available, waiting (3/15)... 2023-07-14T19:17:01.448 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:01 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:01.112+0000 7f3e31dd6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:17:02.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: Activating manager daemon x 2023-07-14T19:17:02.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: mgrmap e2: x(active, starting, since 0.00277882s) 2023-07-14T19:17:02.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T19:17:02.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T19:17:02.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T19:17:02.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:17:02.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T19:17:02.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: Manager daemon x is now available 2023-07-14T19:17:02.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T19:17:02.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T19:17:02.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' 2023-07-14T19:17:02.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' 2023-07-14T19:17:02.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-mon[96565]: from='mgr.14100 172.21.15.53:0/1359334965' entity='mgr.x' 2023-07-14T19:17:02.575 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:02 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:02.165+0000 7f3e31dd6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:17:04.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:04 smithi053 ceph-mon[96565]: mgrmap e3: x(active, since 1.02035s) 2023-07-14T19:17:04.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:17:04.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:17:04.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:17:04.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:17:04.616 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:17:04.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:17:04.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:17:04.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:04.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:17:04.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:17:04.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:17:04.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:17:04.617 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:17:04.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:17:04.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:17:04.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2023-07-14T19:17:04.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:17:04.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:17:04.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:17:04.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:17:04.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:04.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:17:04.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:17:04.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:17:04.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:17:04.626 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:17:04.626 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:17:04.626 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:17:04.626 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:17:04.627 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:04.627 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:17:04.627 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:17:04.627 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:17:04.627 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:17:04.627 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:17:04.627 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:17:04.628 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:17:04.628 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:17:04.628 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:17:04.628 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:04.628 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:17:04.628 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:17:04.629 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:17:04.629 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:17:04.629 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:04.629 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:17:04.629 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T19:17:04.629 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:17:04.630 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:17:04.630 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:17:04.630 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:17:04.630 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:17:04.630 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:17:04.630 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:17:04.631 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:04.631 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:17:04.631 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:17:04.631 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:16:47.795367+0000", 2023-07-14T19:17:04.631 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:17:04.631 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:17:04.631 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:17:04.631 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:17:04.632 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.744+0000 7fce83023700 1 Processor -- start 2023-07-14T19:17:04.632 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.744+0000 7fce83023700 1 -- start start 2023-07-14T19:17:04.632 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.744+0000 7fce83023700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c104830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:04.632 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.744+0000 7fce83023700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c104d70 con 0x7fce7c104420 2023-07-14T19:17:04.632 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.745+0000 7fce80dbf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c104830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:04.632 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.745+0000 7fce80dbf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c104830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48928/0 (socket says 172.21.15.53:48928) 2023-07-14T19:17:04.633 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.745+0000 7fce80dbf700 1 -- 172.21.15.53:0/2041051419 learned_addr learned my addr 172.21.15.53:0/2041051419 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:04.633 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.745+0000 7fce80dbf700 1 -- 172.21.15.53:0/2041051419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7c104eb0 con 0x7fce7c104420 2023-07-14T19:17:04.633 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.745+0000 7fce80dbf700 1 --2- 172.21.15.53:0/2041051419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c104830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce7000bb40 tx=0x7fce7000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989527f1ee90ca02 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:04.633 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.745+0000 7fce7b7fe700 1 -- 172.21.15.53:0/2041051419 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce7000dbe0 con 0x7fce7c104420 2023-07-14T19:17:04.633 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.746+0000 7fce7b7fe700 1 -- 172.21.15.53:0/2041051419 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce700044d0 con 0x7fce7c104420 2023-07-14T19:17:04.633 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.746+0000 7fce7b7fe700 1 -- 172.21.15.53:0/2041051419 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce70010470 con 0x7fce7c104420 2023-07-14T19:17:04.633 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.746+0000 7fce83023700 1 -- 172.21.15.53:0/2041051419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 msgr2=0x7fce7c104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:04.633 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.746+0000 7fce83023700 1 --2- 172.21.15.53:0/2041051419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c104830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce7000bb40 tx=0x7fce7000be50 comp rx=0 tx=0).stop 2023-07-14T19:17:04.634 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.746+0000 7fce83023700 1 -- 172.21.15.53:0/2041051419 shutdown_connections 2023-07-14T19:17:04.634 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.746+0000 7fce83023700 1 --2- 172.21.15.53:0/2041051419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c104830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.634 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.746+0000 7fce83023700 1 -- 172.21.15.53:0/2041051419 >> 172.21.15.53:0/2041051419 conn(0x7fce7c0ffab0 msgr2=0x7fce7c101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:04.634 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.746+0000 7fce83023700 1 -- 172.21.15.53:0/2041051419 shutdown_connections 2023-07-14T19:17:04.634 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.746+0000 7fce83023700 1 -- 172.21.15.53:0/2041051419 wait complete. 2023-07-14T19:17:04.634 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.747+0000 7fce83023700 1 Processor -- start 2023-07-14T19:17:04.634 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.747+0000 7fce83023700 1 -- start start 2023-07-14T19:17:04.634 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.747+0000 7fce83023700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:04.635 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.747+0000 7fce83023700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7c1975d0 con 0x7fce7c104420 2023-07-14T19:17:04.635 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.747+0000 7fce80dbf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:04.635 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.747+0000 7fce80dbf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c197090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48930/0 (socket says 172.21.15.53:48930) 2023-07-14T19:17:04.635 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.747+0000 7fce80dbf700 1 -- 172.21.15.53:0/3251443885 learned_addr learned my addr 172.21.15.53:0/3251443885 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:04.635 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.748+0000 7fce80dbf700 1 -- 172.21.15.53:0/3251443885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7000b7f0 con 0x7fce7c104420 2023-07-14T19:17:04.635 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.748+0000 7fce80dbf700 1 --2- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c197090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fce70003a80 tx=0x7fce70003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:04.636 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.748+0000 7fce79ffb700 1 -- 172.21.15.53:0/3251443885 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce7001e070 con 0x7fce7c104420 2023-07-14T19:17:04.636 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.748+0000 7fce79ffb700 1 -- 172.21.15.53:0/3251443885 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce70004360 con 0x7fce7c104420 2023-07-14T19:17:04.636 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.748+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7c1977d0 con 0x7fce7c104420 2023-07-14T19:17:04.636 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.748+0000 7fce79ffb700 1 -- 172.21.15.53:0/3251443885 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce70022a50 con 0x7fce7c104420 2023-07-14T19:17:04.636 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.748+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7c197bf0 con 0x7fce7c104420 2023-07-14T19:17:04.636 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.749+0000 7fce79ffb700 1 -- 172.21.15.53:0/3251443885 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 3) v1 ==== 45366+0+0 (secure 0 0 0) 0x7fce7002c440 con 0x7fce7c104420 2023-07-14T19:17:04.636 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.749+0000 7fce79ffb700 1 --2- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fce6c038590 0x7fce6c03aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:04.637 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.749+0000 7fce79ffb700 1 -- 172.21.15.53:0/3251443885 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fce7004e8b0 con 0x7fce7c104420 2023-07-14T19:17:04.637 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.749+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce7c190d50 con 0x7fce7c104420 2023-07-14T19:17:04.637 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.749+0000 7fce7bfff700 1 --2- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fce6c038590 0x7fce6c03aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:04.637 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.750+0000 7fce7bfff700 1 --2- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fce6c038590 0x7fce6c03aa40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fce68006fd0 tx=0x7fce68006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:04.637 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:03.755+0000 7fce79ffb700 1 -- 172.21.15.53:0/3251443885 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce70027030 con 0x7fce7c104420 2023-07-14T19:17:04.637 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.195+0000 7fce79ffb700 1 -- 172.21.15.53:0/3251443885 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fce7002c740 con 0x7fce7c104420 2023-07-14T19:17:04.638 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.270+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fce7c061c70 con 0x7fce7c104420 2023-07-14T19:17:04.638 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.270+0000 7fce79ffb700 1 -- 172.21.15.53:0/3251443885 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fce7004c090 con 0x7fce7c104420 2023-07-14T19:17:04.638 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fce6c038590 msgr2=0x7fce6c03aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:04.638 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 --2- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fce6c038590 0x7fce6c03aa40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fce68006fd0 tx=0x7fce68006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:04.638 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 msgr2=0x7fce7c197090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:04.638 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 --2- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c197090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fce70003a80 tx=0x7fce70003df0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.638 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 shutdown_connections 2023-07-14T19:17:04.638 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 --2- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fce6c038590 0x7fce6c03aa40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.639 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 --2- 172.21.15.53:0/3251443885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce7c104420 0x7fce7c197090 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:04.639 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 >> 172.21.15.53:0/3251443885 conn(0x7fce7c0ffab0 msgr2=0x7fce7c18de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:04.639 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 shutdown_connections 2023-07-14T19:17:04.639 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:04.273+0000 7fce83023700 1 -- 172.21.15.53:0/3251443885 wait complete. 2023-07-14T19:17:04.639 INFO:teuthology.orchestra.run.smithi053.stdout:mgr is available 2023-07-14T19:17:05.458 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:05 smithi053 ceph-mon[96565]: mgrmap e4: x(active, since 2s) 2023-07-14T19:17:05.459 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:05 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3251443885' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:17:05.768 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:17:05.769 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T19:17:05.769 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout fsid = c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:17:05.769 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T19:17:05.769 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T19:17:05.769 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T19:17:05.769 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T19:17:05.769 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:17:05.770 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T19:17:05.770 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T19:17:05.770 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2023-07-14T19:17:05.770 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T19:17:05.770 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T19:17:05.770 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T19:17:05.770 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T19:17:05.770 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T19:17:05.771 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T19:17:05.771 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.060+0000 7fcd2a667700 1 Processor -- start 2023-07-14T19:17:05.771 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.060+0000 7fcd2a667700 1 -- start start 2023-07-14T19:17:05.771 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.060+0000 7fcd2a667700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:05.771 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.060+0000 7fcd2a667700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd24105270 con 0x7fcd24104920 2023-07-14T19:17:05.771 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.060+0000 7fcd23fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:05.771 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.061+0000 7fcd23fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48934/0 (socket says 172.21.15.53:48934) 2023-07-14T19:17:05.772 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.061+0000 7fcd23fff700 1 -- 172.21.15.53:0/1847492463 learned_addr learned my addr 172.21.15.53:0/1847492463 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:05.772 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.061+0000 7fcd23fff700 1 -- 172.21.15.53:0/1847492463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd241053b0 con 0x7fcd24104920 2023-07-14T19:17:05.772 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.061+0000 7fcd23fff700 1 --2- 172.21.15.53:0/1847492463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcd140098a0 tx=0x7fcd14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cb622ec00a81b9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:05.772 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.061+0000 7fcd22ffd700 1 -- 172.21.15.53:0/1847492463 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd14004160 con 0x7fcd24104920 2023-07-14T19:17:05.772 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.061+0000 7fcd22ffd700 1 -- 172.21.15.53:0/1847492463 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd140042c0 con 0x7fcd24104920 2023-07-14T19:17:05.773 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.061+0000 7fcd22ffd700 1 -- 172.21.15.53:0/1847492463 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd14011a00 con 0x7fcd24104920 2023-07-14T19:17:05.773 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.062+0000 7fcd2a667700 1 -- 172.21.15.53:0/1847492463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 msgr2=0x7fcd24104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:05.773 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.062+0000 7fcd2a667700 1 --2- 172.21.15.53:0/1847492463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcd140098a0 tx=0x7fcd14009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:05.773 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.062+0000 7fcd2a667700 1 -- 172.21.15.53:0/1847492463 shutdown_connections 2023-07-14T19:17:05.773 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.062+0000 7fcd2a667700 1 --2- 172.21.15.53:0/1847492463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:05.773 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.062+0000 7fcd2a667700 1 -- 172.21.15.53:0/1847492463 >> 172.21.15.53:0/1847492463 conn(0x7fcd240ff210 msgr2=0x7fcd24101620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:05.773 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.062+0000 7fcd2a667700 1 -- 172.21.15.53:0/1847492463 shutdown_connections 2023-07-14T19:17:05.774 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.062+0000 7fcd2a667700 1 -- 172.21.15.53:0/1847492463 wait complete. 2023-07-14T19:17:05.774 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.062+0000 7fcd2a667700 1 Processor -- start 2023-07-14T19:17:05.774 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.063+0000 7fcd2a667700 1 -- start start 2023-07-14T19:17:05.774 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.063+0000 7fcd2a667700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:05.774 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.063+0000 7fcd2a667700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2419c050 con 0x7fcd24104920 2023-07-14T19:17:05.774 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.063+0000 7fcd23fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:05.774 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.063+0000 7fcd23fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48936/0 (socket says 172.21.15.53:48936) 2023-07-14T19:17:05.775 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.063+0000 7fcd23fff700 1 -- 172.21.15.53:0/2629332972 learned_addr learned my addr 172.21.15.53:0/2629332972 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:05.775 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.063+0000 7fcd23fff700 1 -- 172.21.15.53:0/2629332972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd14009550 con 0x7fcd24104920 2023-07-14T19:17:05.775 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.063+0000 7fcd23fff700 1 --2- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd14006af0 tx=0x7fcd14011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:05.775 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.063+0000 7fcd217fa700 1 -- 172.21.15.53:0/2629332972 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd14003a40 con 0x7fcd24104920 2023-07-14T19:17:05.775 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.064+0000 7fcd217fa700 1 -- 172.21.15.53:0/2629332972 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd14003ba0 con 0x7fcd24104920 2023-07-14T19:17:05.775 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.064+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2419c190 con 0x7fcd24104920 2023-07-14T19:17:05.775 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.064+0000 7fcd217fa700 1 -- 172.21.15.53:0/2629332972 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd1401a700 con 0x7fcd24104920 2023-07-14T19:17:05.776 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.064+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2419c5b0 con 0x7fcd24104920 2023-07-14T19:17:05.776 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.065+0000 7fcd217fa700 1 -- 172.21.15.53:0/2629332972 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fcd14024ad0 con 0x7fcd24104920 2023-07-14T19:17:05.776 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.065+0000 7fcd217fa700 1 --2- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fcd100386c0 0x7fcd1003ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:05.777 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.065+0000 7fcd217fa700 1 -- 172.21.15.53:0/2629332972 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcd1404c6b0 con 0x7fcd24104920 2023-07-14T19:17:05.777 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.065+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd241921e0 con 0x7fcd24104920 2023-07-14T19:17:05.777 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.065+0000 7fcd237fe700 1 --2- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fcd100386c0 0x7fcd1003ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:05.778 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.065+0000 7fcd237fe700 1 --2- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fcd100386c0 0x7fcd1003ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:05.778 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.070+0000 7fcd217fa700 1 -- 172.21.15.53:0/2629332972 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcd1402d3f0 con 0x7fcd24104920 2023-07-14T19:17:05.778 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.430+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fcd2402cd00 con 0x7fcd24104920 2023-07-14T19:17:05.778 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.431+0000 7fcd217fa700 1 -- 172.21.15.53:0/2629332972 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fcd140187c0 con 0x7fcd24104920 2023-07-14T19:17:05.779 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fcd100386c0 msgr2=0x7fcd1003ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:05.779 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 --2- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fcd100386c0 0x7fcd1003ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:05.780 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 msgr2=0x7fcd24104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:05.780 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 --2- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd14006af0 tx=0x7fcd14011ad0 comp rx=0 tx=0).stop 2023-07-14T19:17:05.780 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 shutdown_connections 2023-07-14T19:17:05.780 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 --2- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7fcd100386c0 0x7fcd1003ab70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:05.781 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 --2- 172.21.15.53:0/2629332972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd24104920 0x7fcd24104080 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:05.781 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 >> 172.21.15.53:0/2629332972 conn(0x7fcd240ff210 msgr2=0x7fcd24100990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:05.781 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 shutdown_connections 2023-07-14T19:17:05.781 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:05.434+0000 7fcd2a667700 1 -- 172.21.15.53:0/2629332972 wait complete. 2023-07-14T19:17:05.782 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling cephadm module... 2023-07-14T19:17:06.502 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:06 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2629332972' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T19:17:07.486 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:07 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2563705265' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-14T19:17:07.487 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:07.487 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:07.487 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:07.397+0000 7fc0f3be7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:17:07.615 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.255+0000 7f0afa5cb700 1 Processor -- start 2023-07-14T19:17:07.618 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.256+0000 7f0afa5cb700 1 -- start start 2023-07-14T19:17:07.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.256+0000 7f0afa5cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af4073d60 0x7f0af410a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:07.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.256+0000 7f0afa5cb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af4105eb0 con 0x7f0af4073d60 2023-07-14T19:17:07.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.256+0000 7f0af3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af4073d60 0x7f0af410a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:07.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.256+0000 7f0af3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af4073d60 0x7f0af410a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48940/0 (socket says 172.21.15.53:48940) 2023-07-14T19:17:07.619 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.256+0000 7f0af3fff700 1 -- 172.21.15.53:0/3041265649 learned_addr learned my addr 172.21.15.53:0/3041265649 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:07.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.256+0000 7f0af3fff700 1 -- 172.21.15.53:0/3041265649 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af410aa00 con 0x7f0af4073d60 2023-07-14T19:17:07.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.256+0000 7f0af3fff700 1 --2- 172.21.15.53:0/3041265649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af4073d60 0x7f0af410a4c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40098a0 tx=0x7f0ae4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b6fe1becf09d1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:07.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0af2ffd700 1 -- 172.21.15.53:0/3041265649 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ae4004160 con 0x7f0af4073d60 2023-07-14T19:17:07.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0af2ffd700 1 -- 172.21.15.53:0/3041265649 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ae40042c0 con 0x7f0af4073d60 2023-07-14T19:17:07.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0af2ffd700 1 -- 172.21.15.53:0/3041265649 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ae4011a00 con 0x7f0af4073d60 2023-07-14T19:17:07.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0afa5cb700 1 -- 172.21.15.53:0/3041265649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af4073d60 msgr2=0x7f0af410a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:07.620 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0afa5cb700 1 --2- 172.21.15.53:0/3041265649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af4073d60 0x7f0af410a4c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0ae40098a0 tx=0x7f0ae4009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.621 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0afa5cb700 1 -- 172.21.15.53:0/3041265649 shutdown_connections 2023-07-14T19:17:07.621 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0afa5cb700 1 --2- 172.21.15.53:0/3041265649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af4073d60 0x7f0af410a4c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.621 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0afa5cb700 1 -- 172.21.15.53:0/3041265649 >> 172.21.15.53:0/3041265649 conn(0x7f0af40ffab0 msgr2=0x7f0af4101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:07.621 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0afa5cb700 1 -- 172.21.15.53:0/3041265649 shutdown_connections 2023-07-14T19:17:07.621 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.257+0000 7f0afa5cb700 1 -- 172.21.15.53:0/3041265649 wait complete. 2023-07-14T19:17:07.621 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.258+0000 7f0afa5cb700 1 Processor -- start 2023-07-14T19:17:07.621 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.258+0000 7f0afa5cb700 1 -- start start 2023-07-14T19:17:07.621 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0afa5cb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af419b4c0 0x7f0af419b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:07.622 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0afa5cb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af419be10 con 0x7f0af419b4c0 2023-07-14T19:17:07.622 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0af3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af419b4c0 0x7f0af419b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:07.622 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0af3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af419b4c0 0x7f0af419b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48942/0 (socket says 172.21.15.53:48942) 2023-07-14T19:17:07.622 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0af3fff700 1 -- 172.21.15.53:0/2563705265 learned_addr learned my addr 172.21.15.53:0/2563705265 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:07.622 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0af3fff700 1 -- 172.21.15.53:0/2563705265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae4009550 con 0x7f0af419b4c0 2023-07-14T19:17:07.622 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0af3fff700 1 --2- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af419b4c0 0x7f0af419b8d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4006af0 tx=0x7f0ae4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:07.622 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0af17fa700 1 -- 172.21.15.53:0/2563705265 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ae4003a40 con 0x7f0af419b4c0 2023-07-14T19:17:07.623 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0af17fa700 1 -- 172.21.15.53:0/2563705265 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ae4003ba0 con 0x7f0af419b4c0 2023-07-14T19:17:07.623 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0af17fa700 1 -- 172.21.15.53:0/2563705265 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ae401a700 con 0x7f0af419b4c0 2023-07-14T19:17:07.623 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.259+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af419c010 con 0x7f0af419b4c0 2023-07-14T19:17:07.623 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.260+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af419ec70 con 0x7f0af419b4c0 2023-07-14T19:17:07.623 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.260+0000 7f0af17fa700 1 -- 172.21.15.53:0/2563705265 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f0ae4024430 con 0x7f0af419b4c0 2023-07-14T19:17:07.623 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.261+0000 7f0af17fa700 1 --2- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f0adc038710 0x7f0adc03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:07.623 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.261+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af41954f0 con 0x7f0af419b4c0 2023-07-14T19:17:07.623 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.261+0000 7f0af17fa700 1 -- 172.21.15.53:0/2563705265 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0ae404c620 con 0x7f0af419b4c0 2023-07-14T19:17:07.624 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.261+0000 7f0af37fe700 1 --2- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f0adc038710 0x7f0adc03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:07.624 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.261+0000 7f0af37fe700 1 --2- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f0adc038710 0x7f0adc03abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8006fd0 tx=0x7f0ae8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:07.624 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.266+0000 7f0af17fa700 1 -- 172.21.15.53:0/2563705265 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ae4024ad0 con 0x7f0af419b4c0 2023-07-14T19:17:07.624 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:06.736+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f0af4073d60 con 0x7f0af419b4c0 2023-07-14T19:17:07.624 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.202+0000 7f0af17fa700 1 -- 172.21.15.53:0/2563705265 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f0ae402d3f0 con 0x7f0af419b4c0 2023-07-14T19:17:07.624 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.204+0000 7f0af17fa700 1 -- 172.21.15.53:0/2563705265 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f0ae404c040 con 0x7f0af419b4c0 2023-07-14T19:17:07.624 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.207+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f0adc038710 msgr2=0x7f0adc03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:07.625 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.207+0000 7f0afa5cb700 1 --2- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f0adc038710 0x7f0adc03abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8006fd0 tx=0x7f0ae8006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:07.625 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.207+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af419b4c0 msgr2=0x7f0af419b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:07.625 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.207+0000 7f0afa5cb700 1 --2- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af419b4c0 0x7f0af419b8d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4006af0 tx=0x7f0ae4011ad0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.625 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.207+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 shutdown_connections 2023-07-14T19:17:07.625 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.207+0000 7f0afa5cb700 1 --2- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f0adc038710 0x7f0adc03abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.625 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.207+0000 7f0afa5cb700 1 --2- 172.21.15.53:0/2563705265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0af419b4c0 0x7f0af419b8d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:07.625 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.207+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 >> 172.21.15.53:0/2563705265 conn(0x7f0af40ffab0 msgr2=0x7f0af4101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:07.625 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.207+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 shutdown_connections 2023-07-14T19:17:07.626 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:07.208+0000 7f0afa5cb700 1 -- 172.21.15.53:0/2563705265 wait complete. 2023-07-14T19:17:07.822 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:07.496+0000 7fc0f3be7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:17:07.822 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:07.606+0000 7fc0f3be7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:17:07.822 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:07.728+0000 7fc0f3be7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:17:08.204 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:07.945+0000 7fc0f3be7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:17:08.461 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:08 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2563705265' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-14T19:17:08.461 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:08 smithi053 ceph-mon[96565]: mgrmap e5: x(active, since 5s) 2023-07-14T19:17:08.462 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:08.350+0000 7fc0f3be7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:17:08.738 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:08.459+0000 7fc0f3be7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:17:08.738 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:08.612+0000 7fc0f3be7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:17:08.970 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:17:08.972 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-14T19:17:08.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T19:17:08.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T19:17:08.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T19:17:08.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:17:08.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.124+0000 7f8ab5bfa700 1 Processor -- start 2023-07-14T19:17:08.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.124+0000 7f8ab5bfa700 1 -- start start 2023-07-14T19:17:08.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.124+0000 7f8ab5bfa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab0073d60 0x7f8ab010a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:08.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.124+0000 7f8ab5bfa700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab0105eb0 con 0x7f8ab0073d60 2023-07-14T19:17:08.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.124+0000 7f8aaf7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab0073d60 0x7f8ab010a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:08.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.125+0000 7f8aaf7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab0073d60 0x7f8ab010a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48956/0 (socket says 172.21.15.53:48956) 2023-07-14T19:17:08.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.125+0000 7f8aaf7fe700 1 -- 172.21.15.53:0/3185249206 learned_addr learned my addr 172.21.15.53:0/3185249206 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:08.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.125+0000 7f8aaf7fe700 1 -- 172.21.15.53:0/3185249206 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab010aa00 con 0x7f8ab0073d60 2023-07-14T19:17:08.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.125+0000 7f8aaf7fe700 1 --2- 172.21.15.53:0/3185249206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab0073d60 0x7f8ab010a4c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8aa00098a0 tx=0x7f8aa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd03f61b4a185796 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:08.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.125+0000 7f8aae7fc700 1 -- 172.21.15.53:0/3185249206 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8aa0004160 con 0x7f8ab0073d60 2023-07-14T19:17:08.975 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.125+0000 7f8aae7fc700 1 -- 172.21.15.53:0/3185249206 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8aa00042c0 con 0x7f8ab0073d60 2023-07-14T19:17:08.975 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.125+0000 7f8aae7fc700 1 -- 172.21.15.53:0/3185249206 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8aa0011a00 con 0x7f8ab0073d60 2023-07-14T19:17:08.975 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.126+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3185249206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab0073d60 msgr2=0x7f8ab010a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:08.975 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.126+0000 7f8ab5bfa700 1 --2- 172.21.15.53:0/3185249206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab0073d60 0x7f8ab010a4c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8aa00098a0 tx=0x7f8aa0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:08.976 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.126+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3185249206 shutdown_connections 2023-07-14T19:17:08.976 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.126+0000 7f8ab5bfa700 1 --2- 172.21.15.53:0/3185249206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab0073d60 0x7f8ab010a4c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:08.976 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.126+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3185249206 >> 172.21.15.53:0/3185249206 conn(0x7f8ab00ffab0 msgr2=0x7f8ab0101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:08.976 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.126+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3185249206 shutdown_connections 2023-07-14T19:17:08.979 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.126+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3185249206 wait complete. 2023-07-14T19:17:08.979 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.126+0000 7f8ab5bfa700 1 Processor -- start 2023-07-14T19:17:08.979 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.127+0000 7f8ab5bfa700 1 -- start start 2023-07-14T19:17:08.979 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.127+0000 7f8ab5bfa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab019b4c0 0x7f8ab019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:08.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.127+0000 7f8ab5bfa700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab019be10 con 0x7f8ab019b4c0 2023-07-14T19:17:08.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.127+0000 7f8aaf7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab019b4c0 0x7f8ab019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:08.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.127+0000 7f8aaf7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab019b4c0 0x7f8ab019b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48958/0 (socket says 172.21.15.53:48958) 2023-07-14T19:17:08.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.127+0000 7f8aaf7fe700 1 -- 172.21.15.53:0/3181322244 learned_addr learned my addr 172.21.15.53:0/3181322244 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:08.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.127+0000 7f8aaf7fe700 1 -- 172.21.15.53:0/3181322244 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa0009550 con 0x7f8ab019b4c0 2023-07-14T19:17:08.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.127+0000 7f8aaf7fe700 1 --2- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab019b4c0 0x7f8ab019b8d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009fb0 tx=0x7f8aa0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:08.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.128+0000 7f8aacff9700 1 -- 172.21.15.53:0/3181322244 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8aa0003930 con 0x7f8ab019b4c0 2023-07-14T19:17:08.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.128+0000 7f8aacff9700 1 -- 172.21.15.53:0/3181322244 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8aa0003a90 con 0x7f8ab019b4c0 2023-07-14T19:17:08.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.128+0000 7f8aacff9700 1 -- 172.21.15.53:0/3181322244 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8aa00195e0 con 0x7f8ab019b4c0 2023-07-14T19:17:08.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.128+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab019c010 con 0x7f8ab019b4c0 2023-07-14T19:17:08.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.128+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab019ec70 con 0x7f8ab019b4c0 2023-07-14T19:17:08.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.129+0000 7f8aacff9700 1 -- 172.21.15.53:0/3181322244 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f8aa0019850 con 0x7f8ab019b4c0 2023-07-14T19:17:08.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.129+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab01954f0 con 0x7f8ab019b4c0 2023-07-14T19:17:08.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.129+0000 7f8aacff9700 1 --2- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f8a98038710 0x7f8a9803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:08.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.129+0000 7f8aacff9700 1 -- 172.21.15.53:0/3181322244 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8aa004c710 con 0x7f8ab019b4c0 2023-07-14T19:17:08.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.129+0000 7f8aaeffd700 1 -- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f8a98038710 msgr2=0x7f8a9803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3434712469 2023-07-14T19:17:08.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.129+0000 7f8aaeffd700 1 --2- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f8a98038710 0x7f8a9803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:17:08.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.134+0000 7f8aacff9700 1 -- 172.21.15.53:0/3181322244 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8aa002b340 con 0x7f8ab019b4c0 2023-07-14T19:17:08.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.330+0000 7f8aaeffd700 1 -- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f8a98038710 msgr2=0x7f8a9803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3434712469 2023-07-14T19:17:08.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.330+0000 7f8aaeffd700 1 --2- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f8a98038710 0x7f8a9803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:17:08.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.594+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8ab019c1a0 con 0x7f8ab019b4c0 2023-07-14T19:17:08.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.594+0000 7f8aacff9700 1 -- 172.21.15.53:0/3181322244 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f8aa0018b40 con 0x7f8ab019b4c0 2023-07-14T19:17:08.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.596+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f8a98038710 msgr2=0x7f8a9803abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:17:08.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.596+0000 7f8ab5bfa700 1 --2- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f8a98038710 0x7f8a9803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:08.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.596+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab019b4c0 msgr2=0x7f8ab019b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:08.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.596+0000 7f8ab5bfa700 1 --2- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab019b4c0 0x7f8ab019b8d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009fb0 tx=0x7f8aa0011ad0 comp rx=0 tx=0).stop 2023-07-14T19:17:08.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.597+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 shutdown_connections 2023-07-14T19:17:08.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.597+0000 7f8ab5bfa700 1 --2- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f8a98038710 0x7f8a9803abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:08.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.597+0000 7f8ab5bfa700 1 --2- 172.21.15.53:0/3181322244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab019b4c0 0x7f8ab019b8d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:08.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.597+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 >> 172.21.15.53:0/3181322244 conn(0x7f8ab00ffab0 msgr2=0x7f8ab0101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:08.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.597+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 shutdown_connections 2023-07-14T19:17:08.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:08.597+0000 7f8ab5bfa700 1 -- 172.21.15.53:0/3181322244 wait complete. 2023-07-14T19:17:08.985 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for the mgr to restart... 2023-07-14T19:17:08.985 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr epoch 5... 2023-07-14T19:17:09.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:08.739+0000 7fc0f3be7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:17:09.505 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:09 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3181322244' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T19:17:09.505 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:09 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:09.279+0000 7fc0f3be7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:17:10.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:09 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:09.974+0000 7fc0f3be7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:17:10.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:10 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:10.167+0000 7fc0f3be7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:17:11.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:11.134+0000 7fc0f3be7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:17:11.467 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:11.245+0000 7fc0f3be7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:17:11.815 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:11.464+0000 7fc0f3be7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:17:12.070 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:11.813+0000 7fc0f3be7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:17:12.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:12.068+0000 7fc0f3be7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:17:12.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:12.252+0000 7fc0f3be7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:17:12.822 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:12.582+0000 7fc0f3be7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:17:13.821 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:13.413+0000 7fc0f3be7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:17:13.821 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:13.574+0000 7fc0f3be7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:17:14.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:13.819+0000 7fc0f3be7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:17:14.073 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:13.956+0000 7fc0f3be7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:17:14.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:14.229+0000 7fc0f3be7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:17:15.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:15.272+0000 7fc0f3be7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:17:15.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: Active manager daemon x restarted 2023-07-14T19:17:15.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: Activating manager daemon x 2023-07-14T19:17:15.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: osdmap e2: 0 total, 0 up, 0 in 2023-07-14T19:17:15.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: mgrmap e6: x(active, starting, since 0.00313263s) 2023-07-14T19:17:15.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:17:15.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T19:17:15.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T19:17:15.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T19:17:15.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T19:17:15.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: Manager daemon x is now available 2023-07-14T19:17:15.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:15.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:15.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:15.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:15 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:16.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:17:16.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-14T19:17:16.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T19:17:16.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:17:16.651 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.425+0000 7f9bb1cc8700 1 Processor -- start 2023-07-14T19:17:16.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.425+0000 7f9bb1cc8700 1 -- start start 2023-07-14T19:17:16.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.426+0000 7f9bb1cc8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac10a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.426+0000 7f9bb1cc8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac105ed0 con 0x7f9bac079c40 2023-07-14T19:17:16.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.426+0000 7f9bab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac10a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.426+0000 7f9bab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac10a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48966/0 (socket says 172.21.15.53:48966) 2023-07-14T19:17:16.652 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.426+0000 7f9bab7fe700 1 -- 172.21.15.53:0/1737043742 learned_addr learned my addr 172.21.15.53:0/1737043742 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:16.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.426+0000 7f9bab7fe700 1 -- 172.21.15.53:0/1737043742 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac10aa20 con 0x7f9bac079c40 2023-07-14T19:17:16.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.426+0000 7f9bab7fe700 1 --2- 172.21.15.53:0/1737043742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac10a4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0098a0 tx=0x7f9b9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b531e026e645f17 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:16.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.427+0000 7f9baa7fc700 1 -- 172.21.15.53:0/1737043742 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b9c004160 con 0x7f9bac079c40 2023-07-14T19:17:16.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.427+0000 7f9baa7fc700 1 -- 172.21.15.53:0/1737043742 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b9c0042c0 con 0x7f9bac079c40 2023-07-14T19:17:16.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.427+0000 7f9baa7fc700 1 -- 172.21.15.53:0/1737043742 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b9c011a00 con 0x7f9bac079c40 2023-07-14T19:17:16.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.427+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1737043742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 msgr2=0x7f9bac10a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.427+0000 7f9bb1cc8700 1 --2- 172.21.15.53:0/1737043742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac10a4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0098a0 tx=0x7f9b9c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.427+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1737043742 shutdown_connections 2023-07-14T19:17:16.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.427+0000 7f9bb1cc8700 1 --2- 172.21.15.53:0/1737043742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac10a4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.427+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1737043742 >> 172.21.15.53:0/1737043742 conn(0x7f9bac0ffa90 msgr2=0x7f9bac101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:16.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.427+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1737043742 shutdown_connections 2023-07-14T19:17:16.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.428+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1737043742 wait complete. 2023-07-14T19:17:16.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.428+0000 7f9bb1cc8700 1 Processor -- start 2023-07-14T19:17:16.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.428+0000 7f9bb1cc8700 1 -- start start 2023-07-14T19:17:16.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9bb1cc8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac19b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9bb1cc8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac19ba60 con 0x7f9bac079c40 2023-07-14T19:17:16.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9bab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac19b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9bab7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac19b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48968/0 (socket says 172.21.15.53:48968) 2023-07-14T19:17:16.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9bab7fe700 1 -- 172.21.15.53:0/1781478408 learned_addr learned my addr 172.21.15.53:0/1781478408 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:16.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9bab7fe700 1 -- 172.21.15.53:0/1781478408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c009550 con 0x7f9bac079c40 2023-07-14T19:17:16.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9bab7fe700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac19b520 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009fb0 tx=0x7f9b9c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:16.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b9c0039c0 con 0x7f9bac079c40 2023-07-14T19:17:16.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b9c003b20 con 0x7f9bac079c40 2023-07-14T19:17:16.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1781478408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac19bc60 con 0x7f9bac079c40 2023-07-14T19:17:16.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b9c01a780 con 0x7f9bac079c40 2023-07-14T19:17:16.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.429+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1781478408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac19c080 con 0x7f9bac079c40 2023-07-14T19:17:16.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.431+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f9b9c01a8e0 con 0x7f9bac079c40 2023-07-14T19:17:16.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.431+0000 7f9ba8ff9700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 0x7f9b9403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.431+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 --> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b9403b2d0 con 0x7f9b94038710 2023-07-14T19:17:16.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.431+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9b9c04c5d0 con 0x7f9bac079c40 2023-07-14T19:17:16.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.431+0000 7f9baaffd700 1 -- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 msgr2=0x7f9b9403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3434712469 2023-07-14T19:17:16.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.431+0000 7f9baaffd700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 0x7f9b9403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:17:16.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.632+0000 7f9baaffd700 1 -- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 msgr2=0x7f9b9403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3434712469 2023-07-14T19:17:16.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:09.632+0000 7f9baaffd700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 0x7f9b9403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:17:16.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:10.033+0000 7f9baaffd700 1 -- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 msgr2=0x7f9b9403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3434712469 2023-07-14T19:17:16.658 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:10.033+0000 7f9baaffd700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 0x7f9b9403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T19:17:16.658 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:10.834+0000 7f9baaffd700 1 -- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 msgr2=0x7f9b9403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3434712469 2023-07-14T19:17:16.658 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:10.834+0000 7f9baaffd700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 0x7f9b9403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T19:17:16.658 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:12.435+0000 7f9baaffd700 1 -- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 msgr2=0x7f9b9403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/3434712469 2023-07-14T19:17:16.658 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:12.435+0000 7f9baaffd700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 0x7f9b9403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T19:17:16.658 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:15.288+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7f9b9c02d910 con 0x7f9bac079c40 2023-07-14T19:17:16.658 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:15.288+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 msgr2=0x7f9b9403abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:17:16.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:15.288+0000 7f9ba8ff9700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 0x7f9b9403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.289+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f9b9c00fbb0 con 0x7f9bac079c40 2023-07-14T19:17:16.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.289+0000 7f9ba8ff9700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f9b9403bfe0 0x7f9b9403e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:16.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.289+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b9403b2d0 con 0x7f9b9403bfe0 2023-07-14T19:17:16.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.289+0000 7f9baaffd700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f9b9403bfe0 0x7f9b9403e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:16.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.290+0000 7f9baaffd700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f9b9403bfe0 0x7f9b9403e3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0003a10 tx=0x7f9ba00092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:16.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.291+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 <== mgr.14120 v2:172.21.15.53:6800/2149831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f9b9403b2d0 con 0x7f9b9403bfe0 2023-07-14T19:17:16.659 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.304+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1781478408 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9bac19c860 con 0x7f9b9403bfe0 2023-07-14T19:17:16.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9ba8ff9700 1 -- 172.21.15.53:0/1781478408 <== mgr.14120 v2:172.21.15.53:6800/2149831 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9bac19c860 con 0x7f9b9403bfe0 2023-07-14T19:17:16.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f9b9403bfe0 msgr2=0x7f9b9403e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f9b9403bfe0 0x7f9b9403e3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0003a10 tx=0x7f9ba00092b0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 msgr2=0x7f9bac19b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:16.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac19b520 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009fb0 tx=0x7f9b9c011ad0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1781478408 shutdown_connections 2023-07-14T19:17:16.660 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bac079c40 0x7f9bac19b520 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f9b9403bfe0 0x7f9b9403e3c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 --2- 172.21.15.53:0/1781478408 >> [v2:172.21.15.53:6800/3434712469,v1:172.21.15.53:6801/3434712469] conn(0x7f9b94038710 0x7f9b9403abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:16.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1781478408 >> 172.21.15.53:0/1781478408 conn(0x7f9bac0ffa90 msgr2=0x7f9bac1012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:16.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1781478408 shutdown_connections 2023-07-14T19:17:16.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:16.305+0000 7f9bb1cc8700 1 -- 172.21.15.53:0/1781478408 wait complete. 2023-07-14T19:17:16.661 INFO:teuthology.orchestra.run.smithi053.stdout:mgr epoch 5 is available 2023-07-14T19:17:16.661 INFO:teuthology.orchestra.run.smithi053.stdout:Setting orchestrator backend to cephadm... 2023-07-14T19:17:16.662 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:16 smithi053 ceph-mon[96565]: Found migration_current of "None". Setting to last migration. 2023-07-14T19:17:16.662 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:16 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:16.662 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:16 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T19:17:16.663 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:16 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T19:17:16.663 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:16 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:16.663 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:16 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:16.663 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:16 smithi053 ceph-mon[96565]: mgrmap e7: x(active, since 1.00546s) 2023-07-14T19:17:17.670 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:17 smithi053 ceph-mon[96565]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T19:17:17.670 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:17 smithi053 ceph-mon[96565]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T19:17:17.670 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:17 smithi053 ceph-mon[96565]: [14/Jul/2023:19:17:16] ENGINE Bus STARTING 2023-07-14T19:17:17.670 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:17 smithi053 ceph-mon[96565]: [14/Jul/2023:19:17:16] ENGINE Serving on http://172.21.15.53:8765 2023-07-14T19:17:17.670 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:17 smithi053 ceph-mon[96565]: [14/Jul/2023:19:17:16] ENGINE Serving on https://172.21.15.53:7150 2023-07-14T19:17:17.671 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:17 smithi053 ceph-mon[96565]: [14/Jul/2023:19:17:16] ENGINE Bus STARTED 2023-07-14T19:17:17.671 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:17 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.125+0000 7fdf6eccf700 1 Processor -- start 2023-07-14T19:17:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.125+0000 7fdf6eccf700 1 -- start start 2023-07-14T19:17:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.125+0000 7fdf6eccf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68104820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.125+0000 7fdf6eccf700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf68104d60 con 0x7fdf68104410 2023-07-14T19:17:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.125+0000 7fdf6ca6b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68104820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.125+0000 7fdf6ca6b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68104820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49020/0 (socket says 172.21.15.53:49020) 2023-07-14T19:17:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.125+0000 7fdf6ca6b700 1 -- 172.21.15.53:0/4224874716 learned_addr learned my addr 172.21.15.53:0/4224874716 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.126+0000 7fdf6ca6b700 1 -- 172.21.15.53:0/4224874716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf68104ea0 con 0x7fdf68104410 2023-07-14T19:17:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.126+0000 7fdf6ca6b700 1 --2- 172.21.15.53:0/4224874716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68104820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0098a0 tx=0x7fdf5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45d6f5ad3003572 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.126+0000 7fdf677fe700 1 -- 172.21.15.53:0/4224874716 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf5c004160 con 0x7fdf68104410 2023-07-14T19:17:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.126+0000 7fdf677fe700 1 -- 172.21.15.53:0/4224874716 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf5c0042c0 con 0x7fdf68104410 2023-07-14T19:17:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.126+0000 7fdf677fe700 1 -- 172.21.15.53:0/4224874716 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf5c011a00 con 0x7fdf68104410 2023-07-14T19:17:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.126+0000 7fdf6eccf700 1 -- 172.21.15.53:0/4224874716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 msgr2=0x7fdf68104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.126+0000 7fdf6eccf700 1 --2- 172.21.15.53:0/4224874716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68104820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0098a0 tx=0x7fdf5c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.127+0000 7fdf6eccf700 1 -- 172.21.15.53:0/4224874716 shutdown_connections 2023-07-14T19:17:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.127+0000 7fdf6eccf700 1 --2- 172.21.15.53:0/4224874716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68104820 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.127+0000 7fdf6eccf700 1 -- 172.21.15.53:0/4224874716 >> 172.21.15.53:0/4224874716 conn(0x7fdf680ffac0 msgr2=0x7fdf68101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.127+0000 7fdf6eccf700 1 -- 172.21.15.53:0/4224874716 shutdown_connections 2023-07-14T19:17:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.127+0000 7fdf6eccf700 1 -- 172.21.15.53:0/4224874716 wait complete. 2023-07-14T19:17:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.127+0000 7fdf6eccf700 1 Processor -- start 2023-07-14T19:17:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.127+0000 7fdf6eccf700 1 -- start start 2023-07-14T19:17:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.127+0000 7fdf6eccf700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68079af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.127+0000 7fdf6eccf700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6819bfc0 con 0x7fdf68104410 2023-07-14T19:17:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf6ca6b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68079af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf6ca6b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68079af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49022/0 (socket says 172.21.15.53:49022) 2023-07-14T19:17:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf6ca6b700 1 -- 172.21.15.53:0/1562492459 learned_addr learned my addr 172.21.15.53:0/1562492459 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf6ca6b700 1 -- 172.21.15.53:0/1562492459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c009550 con 0x7fdf68104410 2023-07-14T19:17:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf6ca6b700 1 --2- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68079af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c006af0 tx=0x7fdf5c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf65ffb700 1 -- 172.21.15.53:0/1562492459 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf5c003a40 con 0x7fdf68104410 2023-07-14T19:17:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf65ffb700 1 -- 172.21.15.53:0/1562492459 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf5c003ba0 con 0x7fdf68104410 2023-07-14T19:17:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6819c100 con 0x7fdf68104410 2023-07-14T19:17:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf65ffb700 1 -- 172.21.15.53:0/1562492459 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf5c01a6e0 con 0x7fdf68104410 2023-07-14T19:17:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.128+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6819c520 con 0x7fdf68104410 2023-07-14T19:17:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.129+0000 7fdf65ffb700 1 -- 172.21.15.53:0/1562492459 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fdf5c01a950 con 0x7fdf68104410 2023-07-14T19:17:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.130+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf68061c70 con 0x7fdf68104410 2023-07-14T19:17:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.130+0000 7fdf65ffb700 1 --2- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdf580385e0 0x7fdf5803aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.130+0000 7fdf65ffb700 1 -- 172.21.15.53:0/1562492459 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdf5c04c7d0 con 0x7fdf68104410 2023-07-14T19:17:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.130+0000 7fdf67fff700 1 --2- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdf580385e0 0x7fdf5803aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.130+0000 7fdf67fff700 1 --2- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdf580385e0 0x7fdf5803aa90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf54006fd0 tx=0x7fdf54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.135+0000 7fdf65ffb700 1 -- 172.21.15.53:0/1562492459 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdf5c018db0 con 0x7fdf68104410 2023-07-14T19:17:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.521+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf68101de0 con 0x7fdf580385e0 2023-07-14T19:17:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.527+0000 7fdf65ffb700 1 -- 172.21.15.53:0/1562492459 <== mgr.14120 v2:172.21.15.53:6800/2149831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdf68101de0 con 0x7fdf580385e0 2023-07-14T19:17:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdf580385e0 msgr2=0x7fdf5803aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 --2- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdf580385e0 0x7fdf5803aa90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf54006fd0 tx=0x7fdf54006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 msgr2=0x7fdf68079af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 --2- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68079af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c006af0 tx=0x7fdf5c011ad0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 shutdown_connections 2023-07-14T19:17:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 --2- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdf580385e0 0x7fdf5803aa90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 --2- 172.21.15.53:0/1562492459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf68104410 0x7fdf68079af0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 >> 172.21.15.53:0/1562492459 conn(0x7fdf680ffac0 msgr2=0x7fdf681011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 shutdown_connections 2023-07-14T19:17:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:17.529+0000 7fdf6eccf700 1 -- 172.21.15.53:0/1562492459 wait complete. 2023-07-14T19:17:18.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:18 smithi053 ceph-mon[96565]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:18.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:18 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:18.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:18 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:18.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:18 smithi053 ceph-mon[96565]: mgrmap e8: x(active, since 2s) 2023-07-14T19:17:19.140 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-14T19:17:19.140 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.344+0000 7f73f5f6c700 1 Processor -- start 2023-07-14T19:17:19.140 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.344+0000 7f73f5f6c700 1 -- start start 2023-07-14T19:17:19.141 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.344+0000 7f73f5f6c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f0104830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.141 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.344+0000 7f73f5f6c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f0104d70 con 0x7f73f0104420 2023-07-14T19:17:19.141 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.344+0000 7f73ef7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f0104830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.141 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.344+0000 7f73ef7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f0104830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49026/0 (socket says 172.21.15.53:49026) 2023-07-14T19:17:19.141 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.344+0000 7f73ef7fe700 1 -- 172.21.15.53:0/102331091 learned_addr learned my addr 172.21.15.53:0/102331091 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:19.141 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.344+0000 7f73ef7fe700 1 -- 172.21.15.53:0/102331091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f0104eb0 con 0x7f73f0104420 2023-07-14T19:17:19.141 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73ef7fe700 1 --2- 172.21.15.53:0/102331091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f0104830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f73e000bb40 tx=0x7f73e000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770f5f78c84e4659 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.141 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73ee7fc700 1 -- 172.21.15.53:0/102331091 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73e000dbe0 con 0x7f73f0104420 2023-07-14T19:17:19.142 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73ee7fc700 1 -- 172.21.15.53:0/102331091 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73e00044d0 con 0x7f73f0104420 2023-07-14T19:17:19.142 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73ee7fc700 1 -- 172.21.15.53:0/102331091 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73e0010470 con 0x7f73f0104420 2023-07-14T19:17:19.142 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73f5f6c700 1 -- 172.21.15.53:0/102331091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 msgr2=0x7f73f0104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.142 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73f5f6c700 1 --2- 172.21.15.53:0/102331091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f0104830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f73e000bb40 tx=0x7f73e000be50 comp rx=0 tx=0).stop 2023-07-14T19:17:19.142 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73f5f6c700 1 -- 172.21.15.53:0/102331091 shutdown_connections 2023-07-14T19:17:19.142 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73f5f6c700 1 --2- 172.21.15.53:0/102331091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f0104830 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.142 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73f5f6c700 1 -- 172.21.15.53:0/102331091 >> 172.21.15.53:0/102331091 conn(0x7f73f00ffab0 msgr2=0x7f73f0101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.143 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.345+0000 7f73f5f6c700 1 -- 172.21.15.53:0/102331091 shutdown_connections 2023-07-14T19:17:19.143 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.346+0000 7f73f5f6c700 1 -- 172.21.15.53:0/102331091 wait complete. 2023-07-14T19:17:19.143 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.346+0000 7f73f5f6c700 1 Processor -- start 2023-07-14T19:17:19.143 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.346+0000 7f73f5f6c700 1 -- start start 2023-07-14T19:17:19.143 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.346+0000 7f73f5f6c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f019b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.143 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73f5f6c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f019b9e0 con 0x7f73f0104420 2023-07-14T19:17:19.143 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73ef7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f019b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.143 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73ef7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f019b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49028/0 (socket says 172.21.15.53:49028) 2023-07-14T19:17:19.144 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73ef7fe700 1 -- 172.21.15.53:0/1018795501 learned_addr learned my addr 172.21.15.53:0/1018795501 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:19.144 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73ef7fe700 1 -- 172.21.15.53:0/1018795501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e000b7f0 con 0x7f73f0104420 2023-07-14T19:17:19.144 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73ef7fe700 1 --2- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f019b4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f73e000b340 tx=0x7f73e0003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.144 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73ecff9700 1 -- 172.21.15.53:0/1018795501 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73e0004340 con 0x7f73f0104420 2023-07-14T19:17:19.144 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73ecff9700 1 -- 172.21.15.53:0/1018795501 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73e0010bd0 con 0x7f73f0104420 2023-07-14T19:17:19.144 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f019bbe0 con 0x7f73f0104420 2023-07-14T19:17:19.144 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73ecff9700 1 -- 172.21.15.53:0/1018795501 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73e0021c60 con 0x7f73f0104420 2023-07-14T19:17:19.145 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.347+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f019bfa0 con 0x7f73f0104420 2023-07-14T19:17:19.145 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.348+0000 7f73ecff9700 1 -- 172.21.15.53:0/1018795501 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f73e00193f0 con 0x7f73f0104420 2023-07-14T19:17:19.145 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.348+0000 7f73ecff9700 1 --2- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f73d8038710 0x7f73d803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:19.145 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.348+0000 7f73ecff9700 1 -- 172.21.15.53:0/1018795501 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f73e004cb30 con 0x7f73f0104420 2023-07-14T19:17:19.145 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.348+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f0190d50 con 0x7f73f0104420 2023-07-14T19:17:19.145 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.348+0000 7f73eeffd700 1 --2- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f73d8038710 0x7f73d803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:19.146 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.349+0000 7f73eeffd700 1 --2- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f73d8038710 0x7f73d803abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f73e4006fd0 tx=0x7f73e4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:19.146 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.353+0000 7f73ecff9700 1 -- 172.21.15.53:0/1018795501 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f73e000e950 con 0x7f73f0104420 2023-07-14T19:17:19.146 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.711+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f73f0106450 con 0x7f73d8038710 2023-07-14T19:17:19.146 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.712+0000 7f73ecff9700 1 -- 172.21.15.53:0/1018795501 <== mgr.14120 v2:172.21.15.53:6800/2149831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f73f0106450 con 0x7f73d8038710 2023-07-14T19:17:19.146 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.714+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f73d8038710 msgr2=0x7f73d803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.146 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.714+0000 7f73f5f6c700 1 --2- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f73d8038710 0x7f73d803abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f73e4006fd0 tx=0x7f73e4006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:19.147 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.714+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 msgr2=0x7f73f019b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:19.147 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.714+0000 7f73f5f6c700 1 --2- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f019b4a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f73e000b340 tx=0x7f73e0003df0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.147 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.715+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 shutdown_connections 2023-07-14T19:17:19.147 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.715+0000 7f73f5f6c700 1 --2- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f73d8038710 0x7f73d803abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.147 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.715+0000 7f73f5f6c700 1 --2- 172.21.15.53:0/1018795501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73f0104420 0x7f73f019b4a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:19.147 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.715+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 >> 172.21.15.53:0/1018795501 conn(0x7f73f00ffab0 msgr2=0x7f73f018de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:19.147 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.715+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 shutdown_connections 2023-07-14T19:17:19.148 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:18.715+0000 7f73f5f6c700 1 -- 172.21.15.53:0/1018795501 wait complete. 2023-07-14T19:17:19.148 INFO:teuthology.orchestra.run.smithi053.stdout:Generating ssh key... 2023-07-14T19:17:19.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:19 smithi053 ceph-mon[96565]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:20.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: Generating public/private rsa key pair. 2023-07-14T19:17:20.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: Your identification has been saved in /tmp/tmp3vkoc0fo/key. 2023-07-14T19:17:20.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: Your public key has been saved in /tmp/tmp3vkoc0fo/key.pub. 2023-07-14T19:17:20.573 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: The key fingerprint is: 2023-07-14T19:17:20.573 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: SHA256:+vGw77MTyySkkcEHJTOr39EIyuKvs/PZLowZqqsq9AI ceph-c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:17:20.573 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: The key's randomart image is: 2023-07-14T19:17:20.573 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: +---[RSA 3072]----+ 2023-07-14T19:17:20.573 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: | .=o. | 2023-07-14T19:17:20.573 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: | o=. | 2023-07-14T19:17:20.573 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: | o+ | 2023-07-14T19:17:20.574 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: | . oo..o | 2023-07-14T19:17:20.574 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: | . + +S . | 2023-07-14T19:17:20.574 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: |Eo.. ..o..o | 2023-07-14T19:17:20.574 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: |o.o= o ++ o | 2023-07-14T19:17:20.574 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: |o.=ooo . == | 2023-07-14T19:17:20.574 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: |X.+*+oo oo=+ | 2023-07-14T19:17:20.574 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: +----[SHA256]-----+ 2023-07-14T19:17:20.589 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.611+0000 7fcc52ae6700 1 Processor -- start 2023-07-14T19:17:20.590 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.611+0000 7fcc52ae6700 1 -- start start 2023-07-14T19:17:20.590 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.611+0000 7fcc52ae6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c104830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.590 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.611+0000 7fcc52ae6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c104d70 con 0x7fcc4c104420 2023-07-14T19:17:20.590 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.611+0000 7fcc50882700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c104830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.591 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.611+0000 7fcc50882700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c104830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49032/0 (socket says 172.21.15.53:49032) 2023-07-14T19:17:20.591 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.611+0000 7fcc50882700 1 -- 172.21.15.53:0/2484177449 learned_addr learned my addr 172.21.15.53:0/2484177449 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:20.591 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.611+0000 7fcc50882700 1 -- 172.21.15.53:0/2484177449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c104eb0 con 0x7fcc4c104420 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.611+0000 7fcc50882700 1 --2- 172.21.15.53:0/2484177449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c104830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000bb40 tx=0x7fcc4000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6948b494a3aaed0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc4b7fe700 1 -- 172.21.15.53:0/2484177449 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc4000dbe0 con 0x7fcc4c104420 2023-07-14T19:17:20.592 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc4b7fe700 1 -- 172.21.15.53:0/2484177449 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc400044d0 con 0x7fcc4c104420 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc4b7fe700 1 -- 172.21.15.53:0/2484177449 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc40010470 con 0x7fcc4c104420 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2484177449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 msgr2=0x7fcc4c104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.593 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc52ae6700 1 --2- 172.21.15.53:0/2484177449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c104830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000bb40 tx=0x7fcc4000be50 comp rx=0 tx=0).stop 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2484177449 shutdown_connections 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc52ae6700 1 --2- 172.21.15.53:0/2484177449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c104830 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2484177449 >> 172.21.15.53:0/2484177449 conn(0x7fcc4c0ffa90 msgr2=0x7fcc4c101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2484177449 shutdown_connections 2023-07-14T19:17:20.594 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.612+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2484177449 wait complete. 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.613+0000 7fcc52ae6700 1 Processor -- start 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.613+0000 7fcc52ae6700 1 -- start start 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.613+0000 7fcc52ae6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c19b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.595 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.613+0000 7fcc52ae6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c19b9e0 con 0x7fcc4c104420 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.613+0000 7fcc50882700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c19b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.613+0000 7fcc50882700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c19b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49034/0 (socket says 172.21.15.53:49034) 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.613+0000 7fcc50882700 1 -- 172.21.15.53:0/2181777319 learned_addr learned my addr 172.21.15.53:0/2181777319 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:20.596 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.613+0000 7fcc50882700 1 -- 172.21.15.53:0/2181777319 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4000b7f0 con 0x7fcc4c104420 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.614+0000 7fcc50882700 1 --2- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c19b4a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc40003a80 tx=0x7fcc40003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.614+0000 7fcc49ffb700 1 -- 172.21.15.53:0/2181777319 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc400106e0 con 0x7fcc4c104420 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.614+0000 7fcc49ffb700 1 -- 172.21.15.53:0/2181777319 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc40010ce0 con 0x7fcc4c104420 2023-07-14T19:17:20.597 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.614+0000 7fcc49ffb700 1 -- 172.21.15.53:0/2181777319 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc40021c60 con 0x7fcc4c104420 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.614+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c19bbe0 con 0x7fcc4c104420 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.614+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c19bfa0 con 0x7fcc4c104420 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.615+0000 7fcc49ffb700 1 -- 172.21.15.53:0/2181777319 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fcc40021410 con 0x7fcc4c104420 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.615+0000 7fcc49ffb700 1 --2- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fcc3c038760 0x7fcc3c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.615+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c190d50 con 0x7fcc4c104420 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.615+0000 7fcc49ffb700 1 -- 172.21.15.53:0/2181777319 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcc4004ca00 con 0x7fcc4c104420 2023-07-14T19:17:20.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.615+0000 7fcc4bfff700 1 --2- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fcc3c038760 0x7fcc3c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.616+0000 7fcc4bfff700 1 --2- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fcc3c038760 0x7fcc3c03ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcc38006fd0 tx=0x7fcc38006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.620+0000 7fcc49ffb700 1 -- 172.21.15.53:0/2181777319 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcc4000f5b0 con 0x7fcc4c104420 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:19.977+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc4c060a80 con 0x7fcc3c038760 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.241+0000 7fcc49ffb700 1 -- 172.21.15.53:0/2181777319 <== mgr.14120 v2:172.21.15.53:6800/2149831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcc4c060a80 con 0x7fcc3c038760 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.243+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fcc3c038760 msgr2=0x7fcc3c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.243+0000 7fcc52ae6700 1 --2- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fcc3c038760 0x7fcc3c03ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcc38006fd0 tx=0x7fcc38006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:20.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.243+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 msgr2=0x7fcc4c19b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.243+0000 7fcc52ae6700 1 --2- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c19b4a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc40003a80 tx=0x7fcc40003df0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.243+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 shutdown_connections 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.243+0000 7fcc52ae6700 1 --2- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fcc3c038760 0x7fcc3c03ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.243+0000 7fcc52ae6700 1 --2- 172.21.15.53:0/2181777319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc4c104420 0x7fcc4c19b4a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.243+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 >> 172.21.15.53:0/2181777319 conn(0x7fcc4c0ffa90 msgr2=0x7fcc4c18de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.244+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 shutdown_connections 2023-07-14T19:17:20.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:20.244+0000 7fcc52ae6700 1 -- 172.21.15.53:0/2181777319 wait complete. 2023-07-14T19:17:21.544 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:21 smithi053 ceph-mon[96565]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:21.545 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:21 smithi053 ceph-mon[96565]: Generating ssh key... 2023-07-14T19:17:21.545 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:21 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:21.545 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:21 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:21.735 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChapZxBNZKdrzqvQpUT/4xiLDyvur3Bnvzfj8qJCG6gZVdrHaT6wbHXlGQxAX4f3bUzERuKe+yAS+uMKC4TOJP8JPh8kbbTJkQHczIEydEGXUsZe/H4kjrFsTD/pfS0KcRlwSAs1eLWZcJ3nAW+l2BPuaiGf1hTItTZ5T2DhDmYGFMEobg4xze24Fpoxes0PiZZFJ/jKsMku4G0kIqK6o8V4csVhf0SBDM5t+LHTD5jMBLlt/p0LCfQ9f39N6y0cwNVoF+ZGivCkASannUt6np/EFQBIS9Q08hd9zXQnstYsdeMUAOoq/XsQpaxDR+K9LCLfs8zPEAvKfgRGyfIPO3tOk75oPthF0wzYsHKfd4AQpw6xXiK1qLyAcnYTtl8Ri6In+k/EopYbWnd8S57ZhOWInYP7vBbrdsUP2yMPQKCCagkZSTQJ0HeGRMFsoBJjR8LKkCcRnCssiMO+AAMABcJjOxpqmJxoRn5dgS7oI+q1s8Z2CJ627jXhxpvkifeLs= ceph-c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:17:21.735 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.029+0000 7f2f257e6700 1 Processor -- start 2023-07-14T19:17:21.735 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.029+0000 7f2f257e6700 1 -- start start 2023-07-14T19:17:21.737 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.029+0000 7f2f257e6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2010a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:21.737 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.029+0000 7f2f257e6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f20105ec0 con 0x7f2f20079c40 2023-07-14T19:17:21.738 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.030+0000 7f2f1effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2010a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:21.738 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.030+0000 7f2f1effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2010a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49038/0 (socket says 172.21.15.53:49038) 2023-07-14T19:17:21.738 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.030+0000 7f2f1effd700 1 -- 172.21.15.53:0/1329088294 learned_addr learned my addr 172.21.15.53:0/1329088294 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:21.738 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.030+0000 7f2f1effd700 1 -- 172.21.15.53:0/1329088294 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2010aa10 con 0x7f2f20079c40 2023-07-14T19:17:21.738 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.030+0000 7f2f1effd700 1 --2- 172.21.15.53:0/1329088294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2010a4d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2f100098a0 tx=0x7f2f10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b5e5c4052b12f9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:21.738 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.030+0000 7f2f1dffb700 1 -- 172.21.15.53:0/1329088294 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f10004160 con 0x7f2f20079c40 2023-07-14T19:17:21.738 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.030+0000 7f2f1dffb700 1 -- 172.21.15.53:0/1329088294 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f100042c0 con 0x7f2f20079c40 2023-07-14T19:17:21.738 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.030+0000 7f2f1dffb700 1 -- 172.21.15.53:0/1329088294 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f10011a00 con 0x7f2f20079c40 2023-07-14T19:17:21.739 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.031+0000 7f2f257e6700 1 -- 172.21.15.53:0/1329088294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 msgr2=0x7f2f2010a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:21.739 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.031+0000 7f2f257e6700 1 --2- 172.21.15.53:0/1329088294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2010a4d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2f100098a0 tx=0x7f2f10009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:21.739 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.031+0000 7f2f257e6700 1 -- 172.21.15.53:0/1329088294 shutdown_connections 2023-07-14T19:17:21.739 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.031+0000 7f2f257e6700 1 --2- 172.21.15.53:0/1329088294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2010a4d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:21.739 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.031+0000 7f2f257e6700 1 -- 172.21.15.53:0/1329088294 >> 172.21.15.53:0/1329088294 conn(0x7f2f200ffac0 msgr2=0x7f2f20101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:21.739 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.031+0000 7f2f257e6700 1 -- 172.21.15.53:0/1329088294 shutdown_connections 2023-07-14T19:17:21.739 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.031+0000 7f2f257e6700 1 -- 172.21.15.53:0/1329088294 wait complete. 2023-07-14T19:17:21.740 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.032+0000 7f2f257e6700 1 Processor -- start 2023-07-14T19:17:21.740 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.032+0000 7f2f257e6700 1 -- start start 2023-07-14T19:17:21.740 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.032+0000 7f2f257e6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:21.740 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.032+0000 7f2f257e6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2019ba40 con 0x7f2f20079c40 2023-07-14T19:17:21.740 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.032+0000 7f2f1effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:21.741 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.032+0000 7f2f1effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2019b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49040/0 (socket says 172.21.15.53:49040) 2023-07-14T19:17:21.741 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.032+0000 7f2f1effd700 1 -- 172.21.15.53:0/123402636 learned_addr learned my addr 172.21.15.53:0/123402636 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:21.741 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.032+0000 7f2f1effd700 1 -- 172.21.15.53:0/123402636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f10009550 con 0x7f2f20079c40 2023-07-14T19:17:21.741 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.033+0000 7f2f1effd700 1 --2- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2019b500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009fb0 tx=0x7f2f10011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:21.741 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.033+0000 7f2f07fff700 1 -- 172.21.15.53:0/123402636 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f100039c0 con 0x7f2f20079c40 2023-07-14T19:17:21.741 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.033+0000 7f2f07fff700 1 -- 172.21.15.53:0/123402636 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f10003b20 con 0x7f2f20079c40 2023-07-14T19:17:21.741 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.033+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2019bc40 con 0x7f2f20079c40 2023-07-14T19:17:21.741 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.033+0000 7f2f07fff700 1 -- 172.21.15.53:0/123402636 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f1001a780 con 0x7f2f20079c40 2023-07-14T19:17:21.742 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.033+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2019c060 con 0x7f2f20079c40 2023-07-14T19:17:21.742 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.034+0000 7f2f07fff700 1 -- 172.21.15.53:0/123402636 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f2f1001a8e0 con 0x7f2f20079c40 2023-07-14T19:17:21.742 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.034+0000 7f2f07fff700 1 --2- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f2f08038710 0x7f2f0803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:21.742 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.034+0000 7f2f07fff700 1 -- 172.21.15.53:0/123402636 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2f1004c970 con 0x7f2f20079c40 2023-07-14T19:17:21.742 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.034+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f201954e0 con 0x7f2f20079c40 2023-07-14T19:17:21.742 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.034+0000 7f2f1e7fc700 1 --2- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f2f08038710 0x7f2f0803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:21.742 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.034+0000 7f2f1e7fc700 1 --2- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f2f08038710 0x7f2f0803abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2f14006fd0 tx=0x7f2f14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:21.743 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.039+0000 7f2f07fff700 1 -- 172.21.15.53:0/123402636 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2f1001abe0 con 0x7f2f20079c40 2023-07-14T19:17:21.743 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.398+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f20060a80 con 0x7f2f08038710 2023-07-14T19:17:21.743 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.399+0000 7f2f07fff700 1 -- 172.21.15.53:0/123402636 <== mgr.14120 v2:172.21.15.53:6800/2149831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f2f20060a80 con 0x7f2f08038710 2023-07-14T19:17:21.743 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f2f08038710 msgr2=0x7f2f0803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:21.743 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 --2- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f2f08038710 0x7f2f0803abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2f14006fd0 tx=0x7f2f14006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:21.743 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 msgr2=0x7f2f2019b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:21.743 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 --2- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2019b500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2f10009fb0 tx=0x7f2f10011ad0 comp rx=0 tx=0).stop 2023-07-14T19:17:21.744 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 shutdown_connections 2023-07-14T19:17:21.744 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 --2- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f2f08038710 0x7f2f0803abc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:21.744 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 --2- 172.21.15.53:0/123402636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f20079c40 0x7f2f2019b500 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:21.744 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 >> 172.21.15.53:0/123402636 conn(0x7f2f200ffac0 msgr2=0x7f2f201012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:21.744 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 shutdown_connections 2023-07-14T19:17:21.744 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:21.401+0000 7f2f257e6700 1 -- 172.21.15.53:0/123402636 wait complete. 2023-07-14T19:17:21.745 INFO:teuthology.orchestra.run.smithi053.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:17:21.745 INFO:teuthology.orchestra.run.smithi053.stdout:Adding key to root@localhost authorized_keys... 2023-07-14T19:17:21.746 INFO:teuthology.orchestra.run.smithi053.stdout:Adding host smithi053... 2023-07-14T19:17:23.404 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:23 smithi053 ceph-mon[96565]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:24.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:24 smithi053 ceph-mon[96565]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:24.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:24 smithi053 ceph-mon[96565]: Deploying cephadm binary to smithi053 2023-07-14T19:17:25.990 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Added host 'smithi053' with addr '172.21.15.53' 2023-07-14T19:17:25.990 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.206+0000 7eff1f90c700 1 Processor -- start 2023-07-14T19:17:25.990 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.207+0000 7eff1f90c700 1 -- start start 2023-07-14T19:17:25.990 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.207+0000 7eff1f90c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff18104820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.990 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.207+0000 7eff1f90c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff18104d60 con 0x7eff18104410 2023-07-14T19:17:25.991 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.207+0000 7eff1d6a8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff18104820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.991 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.207+0000 7eff1d6a8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff18104820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49044/0 (socket says 172.21.15.53:49044) 2023-07-14T19:17:25.991 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.207+0000 7eff1d6a8700 1 -- 172.21.15.53:0/349306955 learned_addr learned my addr 172.21.15.53:0/349306955 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:25.991 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.207+0000 7eff1d6a8700 1 -- 172.21.15.53:0/349306955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff18104ea0 con 0x7eff18104410 2023-07-14T19:17:25.991 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.207+0000 7eff1d6a8700 1 --2- 172.21.15.53:0/349306955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff18104820 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff140098a0 tx=0x7eff14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608227cb7ffe4e90 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:25.991 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.208+0000 7eff0ffff700 1 -- 172.21.15.53:0/349306955 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff14004160 con 0x7eff18104410 2023-07-14T19:17:25.991 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.208+0000 7eff0ffff700 1 -- 172.21.15.53:0/349306955 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff140042c0 con 0x7eff18104410 2023-07-14T19:17:25.992 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.208+0000 7eff0ffff700 1 -- 172.21.15.53:0/349306955 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff14011a00 con 0x7eff18104410 2023-07-14T19:17:25.992 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.208+0000 7eff1f90c700 1 -- 172.21.15.53:0/349306955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 msgr2=0x7eff18104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.992 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.208+0000 7eff1f90c700 1 --2- 172.21.15.53:0/349306955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff18104820 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff140098a0 tx=0x7eff14009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.992 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.208+0000 7eff1f90c700 1 -- 172.21.15.53:0/349306955 shutdown_connections 2023-07-14T19:17:25.992 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.208+0000 7eff1f90c700 1 --2- 172.21.15.53:0/349306955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff18104820 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.992 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.208+0000 7eff1f90c700 1 -- 172.21.15.53:0/349306955 >> 172.21.15.53:0/349306955 conn(0x7eff180ffac0 msgr2=0x7eff18101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:25.992 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.208+0000 7eff1f90c700 1 -- 172.21.15.53:0/349306955 shutdown_connections 2023-07-14T19:17:25.993 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.209+0000 7eff1f90c700 1 -- 172.21.15.53:0/349306955 wait complete. 2023-07-14T19:17:25.993 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.209+0000 7eff1f90c700 1 Processor -- start 2023-07-14T19:17:25.993 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.209+0000 7eff1f90c700 1 -- start start 2023-07-14T19:17:25.993 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.209+0000 7eff1f90c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff181992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.993 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.209+0000 7eff1f90c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff18199810 con 0x7eff18104410 2023-07-14T19:17:25.993 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.209+0000 7eff1d6a8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff181992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.993 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.209+0000 7eff1d6a8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff181992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49046/0 (socket says 172.21.15.53:49046) 2023-07-14T19:17:25.993 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.209+0000 7eff1d6a8700 1 -- 172.21.15.53:0/1024979070 learned_addr learned my addr 172.21.15.53:0/1024979070 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:25.994 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.210+0000 7eff1d6a8700 1 -- 172.21.15.53:0/1024979070 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff14009550 con 0x7eff18104410 2023-07-14T19:17:25.994 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.210+0000 7eff1d6a8700 1 --2- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff181992d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff14009fb0 tx=0x7eff14011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:25.994 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.210+0000 7eff0e7fc700 1 -- 172.21.15.53:0/1024979070 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff14003930 con 0x7eff18104410 2023-07-14T19:17:25.994 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.210+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff18199a10 con 0x7eff18104410 2023-07-14T19:17:25.994 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.210+0000 7eff0e7fc700 1 -- 172.21.15.53:0/1024979070 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff14003a90 con 0x7eff18104410 2023-07-14T19:17:25.994 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.210+0000 7eff0e7fc700 1 -- 172.21.15.53:0/1024979070 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff14019610 con 0x7eff18104410 2023-07-14T19:17:25.994 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.210+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff18199e30 con 0x7eff18104410 2023-07-14T19:17:25.994 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.211+0000 7eff0e7fc700 1 -- 172.21.15.53:0/1024979070 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7eff1401a9d0 con 0x7eff18104410 2023-07-14T19:17:25.995 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.211+0000 7eff0e7fc700 1 --2- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7eff040386c0 0x7eff0403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:25.995 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.211+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff18061c70 con 0x7eff18104410 2023-07-14T19:17:25.995 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.211+0000 7eff0e7fc700 1 -- 172.21.15.53:0/1024979070 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7eff1404c7e0 con 0x7eff18104410 2023-07-14T19:17:25.995 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.211+0000 7eff1cea7700 1 --2- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7eff040386c0 0x7eff0403ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:25.995 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.212+0000 7eff1cea7700 1 --2- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7eff040386c0 0x7eff0403ab70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7eff08006fd0 tx=0x7eff08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:25.995 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.216+0000 7eff0e7fc700 1 -- 172.21.15.53:0/1024979070 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7eff140198d0 con 0x7eff18104410 2023-07-14T19:17:25.995 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:22.580+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}) v1 -- 0x7eff18101de0 con 0x7eff040386c0 2023-07-14T19:17:25.996 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.566+0000 7eff0e7fc700 1 -- 172.21.15.53:0/1024979070 <== mgr.14120 v2:172.21.15.53:6800/2149831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7eff18101de0 con 0x7eff040386c0 2023-07-14T19:17:25.996 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7eff040386c0 msgr2=0x7eff0403ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.996 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 --2- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7eff040386c0 0x7eff0403ab70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7eff08006fd0 tx=0x7eff08006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:25.996 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 msgr2=0x7eff181992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:25.996 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 --2- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff181992d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff14009fb0 tx=0x7eff14011ad0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.996 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 shutdown_connections 2023-07-14T19:17:25.996 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 --2- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7eff040386c0 0x7eff0403ab70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.997 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 --2- 172.21.15.53:0/1024979070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff18104410 0x7eff181992d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:25.997 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 >> 172.21.15.53:0/1024979070 conn(0x7eff180ffac0 msgr2=0x7eff181011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:25.997 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 shutdown_connections 2023-07-14T19:17:25.997 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:25.569+0000 7eff1f90c700 1 -- 172.21.15.53:0/1024979070 wait complete. 2023-07-14T19:17:25.997 INFO:teuthology.orchestra.run.smithi053.stdout:Deploying unmanaged mon service... 2023-07-14T19:17:26.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:26 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:26.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:26 smithi053 ceph-mon[96565]: Added host smithi053 2023-07-14T19:17:26.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:26 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:27.442 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-14T19:17:27.443 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.484+0000 7f36df7fe700 1 Processor -- start 2023-07-14T19:17:27.443 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.484+0000 7f36df7fe700 1 -- start start 2023-07-14T19:17:27.443 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.484+0000 7f36df7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8108410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.443 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.484+0000 7f36df7fe700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d8073de0 con 0x7f36d8106030 2023-07-14T19:17:27.443 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.485+0000 7f36dd59a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.443 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.485+0000 7f36dd59a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8108410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49080/0 (socket says 172.21.15.53:49080) 2023-07-14T19:17:27.443 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.485+0000 7f36dd59a700 1 -- 172.21.15.53:0/2107546560 learned_addr learned my addr 172.21.15.53:0/2107546560 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:27.444 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.485+0000 7f36dd59a700 1 -- 172.21.15.53:0/2107546560 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8073f20 con 0x7f36d8106030 2023-07-14T19:17:27.444 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.485+0000 7f36dd59a700 1 --2- 172.21.15.53:0/2107546560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8108410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f36c80098a0 tx=0x7f36c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c724a8ddd3058549 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:27.444 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.485+0000 7f36cffff700 1 -- 172.21.15.53:0/2107546560 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36c8004160 con 0x7f36d8106030 2023-07-14T19:17:27.444 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.485+0000 7f36cffff700 1 -- 172.21.15.53:0/2107546560 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36c80042c0 con 0x7f36d8106030 2023-07-14T19:17:27.444 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.485+0000 7f36cffff700 1 -- 172.21.15.53:0/2107546560 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36c8011a00 con 0x7f36d8106030 2023-07-14T19:17:27.444 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.486+0000 7f36df7fe700 1 -- 172.21.15.53:0/2107546560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 msgr2=0x7f36d8108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.444 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.486+0000 7f36df7fe700 1 --2- 172.21.15.53:0/2107546560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8108410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f36c80098a0 tx=0x7f36c8009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.445 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.486+0000 7f36df7fe700 1 -- 172.21.15.53:0/2107546560 shutdown_connections 2023-07-14T19:17:27.445 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.486+0000 7f36df7fe700 1 --2- 172.21.15.53:0/2107546560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8108410 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.445 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.486+0000 7f36df7fe700 1 -- 172.21.15.53:0/2107546560 >> 172.21.15.53:0/2107546560 conn(0x7f36d80ffab0 msgr2=0x7f36d8101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:27.445 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.486+0000 7f36df7fe700 1 -- 172.21.15.53:0/2107546560 shutdown_connections 2023-07-14T19:17:27.445 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.486+0000 7f36df7fe700 1 -- 172.21.15.53:0/2107546560 wait complete. 2023-07-14T19:17:27.445 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.487+0000 7f36df7fe700 1 Processor -- start 2023-07-14T19:17:27.445 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.487+0000 7f36df7fe700 1 -- start start 2023-07-14T19:17:27.445 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.487+0000 7f36df7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8105990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.446 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.487+0000 7f36df7fe700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d8105ed0 con 0x7f36d8106030 2023-07-14T19:17:27.446 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.487+0000 7f36dd59a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8105990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.446 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.487+0000 7f36dd59a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8105990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49082/0 (socket says 172.21.15.53:49082) 2023-07-14T19:17:27.446 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.487+0000 7f36dd59a700 1 -- 172.21.15.53:0/377429056 learned_addr learned my addr 172.21.15.53:0/377429056 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:27.446 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.487+0000 7f36dd59a700 1 -- 172.21.15.53:0/377429056 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c8009550 con 0x7f36d8106030 2023-07-14T19:17:27.446 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.488+0000 7f36dd59a700 1 --2- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8105990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f36c8006af0 tx=0x7f36c8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:27.446 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.488+0000 7f36ce7fc700 1 -- 172.21.15.53:0/377429056 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36c8003a40 con 0x7f36d8106030 2023-07-14T19:17:27.447 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.488+0000 7f36ce7fc700 1 -- 172.21.15.53:0/377429056 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36c8003ba0 con 0x7f36d8106030 2023-07-14T19:17:27.447 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.488+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d8103fe0 con 0x7f36d8106030 2023-07-14T19:17:27.447 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.488+0000 7f36ce7fc700 1 -- 172.21.15.53:0/377429056 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36c801a700 con 0x7f36d8106030 2023-07-14T19:17:27.447 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.488+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d8104400 con 0x7f36d8106030 2023-07-14T19:17:27.447 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.489+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d8109620 con 0x7f36d8106030 2023-07-14T19:17:27.447 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.489+0000 7f36ce7fc700 1 -- 172.21.15.53:0/377429056 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f36c8024ad0 con 0x7f36d8106030 2023-07-14T19:17:27.447 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.489+0000 7f36ce7fc700 1 --2- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f36c4038760 0x7f36c403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:27.447 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.489+0000 7f36ce7fc700 1 -- 172.21.15.53:0/377429056 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f36c804c6b0 con 0x7f36d8106030 2023-07-14T19:17:27.448 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.489+0000 7f36dcd99700 1 --2- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f36c4038760 0x7f36c403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:27.448 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.490+0000 7f36dcd99700 1 --2- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f36c4038760 0x7f36c403ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f36d4006fd0 tx=0x7f36d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:27.448 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.494+0000 7f36ce7fc700 1 -- 172.21.15.53:0/377429056 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f36c8018b80 con 0x7f36d8106030 2023-07-14T19:17:27.448 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.897+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f36d8101df0 con 0x7f36c4038760 2023-07-14T19:17:27.448 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.903+0000 7f36ce7fc700 1 -- 172.21.15.53:0/377429056 <== mgr.14120 v2:172.21.15.53:6800/2149831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f36d8101df0 con 0x7f36c4038760 2023-07-14T19:17:27.448 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.905+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f36c4038760 msgr2=0x7f36c403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.448 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.905+0000 7f36df7fe700 1 --2- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f36c4038760 0x7f36c403ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f36d4006fd0 tx=0x7f36d4006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:27.449 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.905+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 msgr2=0x7f36d8105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:27.449 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.905+0000 7f36df7fe700 1 --2- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8105990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f36c8006af0 tx=0x7f36c8011ad0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.449 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.905+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 shutdown_connections 2023-07-14T19:17:27.449 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.905+0000 7f36df7fe700 1 --2- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f36c4038760 0x7f36c403ac10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.449 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.905+0000 7f36df7fe700 1 --2- 172.21.15.53:0/377429056 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36d8106030 0x7f36d8105990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:27.449 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.906+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 >> 172.21.15.53:0/377429056 conn(0x7f36d80ffab0 msgr2=0x7f36d8101190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:27.449 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.906+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 shutdown_connections 2023-07-14T19:17:27.449 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:26.906+0000 7f36df7fe700 1 -- 172.21.15.53:0/377429056 wait complete. 2023-07-14T19:17:27.450 INFO:teuthology.orchestra.run.smithi053.stdout:Deploying unmanaged mgr service... 2023-07-14T19:17:28.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:27 smithi053 ceph-mon[96565]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:28.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:27 smithi053 ceph-mon[96565]: Saving service mon spec with placement count:5 2023-07-14T19:17:28.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:27 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:28.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:27 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:28.966 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-14T19:17:28.966 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.157+0000 7ff2b184e700 1 Processor -- start 2023-07-14T19:17:28.966 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.157+0000 7ff2b184e700 1 -- start start 2023-07-14T19:17:28.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.157+0000 7ff2b184e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac104820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.157+0000 7ff2b184e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac104d60 con 0x7ff2ac104410 2023-07-14T19:17:28.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.157+0000 7ff2aaffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac104820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.157+0000 7ff2aaffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac104820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49086/0 (socket says 172.21.15.53:49086) 2023-07-14T19:17:28.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.157+0000 7ff2aaffd700 1 -- 172.21.15.53:0/849281607 learned_addr learned my addr 172.21.15.53:0/849281607 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:28.967 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.158+0000 7ff2aaffd700 1 -- 172.21.15.53:0/849281607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac104ea0 con 0x7ff2ac104410 2023-07-14T19:17:28.968 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.158+0000 7ff2aaffd700 1 --2- 172.21.15.53:0/849281607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac104820 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2940098a0 tx=0x7ff294009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1856d4e17844c6c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:28.968 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.158+0000 7ff2a9ffb700 1 -- 172.21.15.53:0/849281607 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff294004160 con 0x7ff2ac104410 2023-07-14T19:17:28.968 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.158+0000 7ff2a9ffb700 1 -- 172.21.15.53:0/849281607 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2940042c0 con 0x7ff2ac104410 2023-07-14T19:17:28.968 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.158+0000 7ff2a9ffb700 1 -- 172.21.15.53:0/849281607 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2940119b0 con 0x7ff2ac104410 2023-07-14T19:17:28.968 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.159+0000 7ff2b184e700 1 -- 172.21.15.53:0/849281607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 msgr2=0x7ff2ac104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.968 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.159+0000 7ff2b184e700 1 --2- 172.21.15.53:0/849281607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac104820 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2940098a0 tx=0x7ff294009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.969 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.159+0000 7ff2b184e700 1 -- 172.21.15.53:0/849281607 shutdown_connections 2023-07-14T19:17:28.969 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.159+0000 7ff2b184e700 1 --2- 172.21.15.53:0/849281607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac104820 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.969 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.159+0000 7ff2b184e700 1 -- 172.21.15.53:0/849281607 >> 172.21.15.53:0/849281607 conn(0x7ff2ac0ffac0 msgr2=0x7ff2ac101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:28.969 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.159+0000 7ff2b184e700 1 -- 172.21.15.53:0/849281607 shutdown_connections 2023-07-14T19:17:28.969 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.159+0000 7ff2b184e700 1 -- 172.21.15.53:0/849281607 wait complete. 2023-07-14T19:17:28.969 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.159+0000 7ff2b184e700 1 Processor -- start 2023-07-14T19:17:28.971 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.160+0000 7ff2b184e700 1 -- start start 2023-07-14T19:17:28.971 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.160+0000 7ff2b184e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac079af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.971 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.160+0000 7ff2b184e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac19bfe0 con 0x7ff2ac104410 2023-07-14T19:17:28.971 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.160+0000 7ff2aaffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac079af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.972 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.160+0000 7ff2aaffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac079af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49088/0 (socket says 172.21.15.53:49088) 2023-07-14T19:17:28.972 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.160+0000 7ff2aaffd700 1 -- 172.21.15.53:0/1531319226 learned_addr learned my addr 172.21.15.53:0/1531319226 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:28.972 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.160+0000 7ff2aaffd700 1 -- 172.21.15.53:0/1531319226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff294009550 con 0x7ff2ac104410 2023-07-14T19:17:28.972 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.160+0000 7ff2aaffd700 1 --2- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac079af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff294003e60 tx=0x7ff294011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:28.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.162+0000 7ff2b084c700 1 -- 172.21.15.53:0/1531319226 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff294003970 con 0x7ff2ac104410 2023-07-14T19:17:28.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.162+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac19c120 con 0x7ff2ac104410 2023-07-14T19:17:28.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.162+0000 7ff2b084c700 1 -- 172.21.15.53:0/1531319226 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff294003ad0 con 0x7ff2ac104410 2023-07-14T19:17:28.973 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.162+0000 7ff2b084c700 1 -- 172.21.15.53:0/1531319226 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2940195e0 con 0x7ff2ac104410 2023-07-14T19:17:28.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.162+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ac19c540 con 0x7ff2ac104410 2023-07-14T19:17:28.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.163+0000 7ff2b084c700 1 -- 172.21.15.53:0/1531319226 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff2940198a0 con 0x7ff2ac104410 2023-07-14T19:17:28.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.163+0000 7ff2b084c700 1 --2- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7ff2980386c0 0x7ff29803ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:28.974 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.163+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28c005320 con 0x7ff2ac104410 2023-07-14T19:17:28.975 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.163+0000 7ff2b084c700 1 -- 172.21.15.53:0/1531319226 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff29404d840 con 0x7ff2ac104410 2023-07-14T19:17:28.975 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.163+0000 7ff2aa7fc700 1 --2- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7ff2980386c0 0x7ff29803ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:28.975 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.164+0000 7ff2aa7fc700 1 --2- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7ff2980386c0 0x7ff29803ab70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff29c006fd0 tx=0x7ff29c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:28.975 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.168+0000 7ff2b084c700 1 -- 172.21.15.53:0/1531319226 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff29402b340 con 0x7ff2ac104410 2023-07-14T19:17:28.976 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.556+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff28c000bf0 con 0x7ff2980386c0 2023-07-14T19:17:28.976 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.560+0000 7ff2b084c700 1 -- 172.21.15.53:0/1531319226 <== mgr.14120 v2:172.21.15.53:6800/2149831 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff28c000bf0 con 0x7ff2980386c0 2023-07-14T19:17:28.976 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7ff2980386c0 msgr2=0x7ff29803ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.976 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 --2- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7ff2980386c0 0x7ff29803ab70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff29c006fd0 tx=0x7ff29c006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:28.977 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 msgr2=0x7ff2ac079af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:28.977 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 --2- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac079af0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff294003e60 tx=0x7ff294011aa0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.977 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 shutdown_connections 2023-07-14T19:17:28.977 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 --2- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7ff2980386c0 0x7ff29803ab70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.977 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 --2- 172.21.15.53:0/1531319226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2ac104410 0x7ff2ac079af0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:28.978 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 >> 172.21.15.53:0/1531319226 conn(0x7ff2ac0ffac0 msgr2=0x7ff2ac1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:28.978 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 shutdown_connections 2023-07-14T19:17:28.978 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:28.563+0000 7ff2b184e700 1 -- 172.21.15.53:0/1531319226 wait complete. 2023-07-14T19:17:29.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:29 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:29.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:29 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:30.210 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.425+0000 7f163bd6d700 1 Processor -- start 2023-07-14T19:17:30.211 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.425+0000 7f163bd6d700 1 -- start start 2023-07-14T19:17:30.211 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.426+0000 7f163bd6d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f1634108440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:30.211 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.426+0000 7f163bd6d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1634079ca0 con 0x7f1634106060 2023-07-14T19:17:30.211 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.426+0000 7f1639b09700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f1634108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:30.211 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.426+0000 7f1639b09700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f1634108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49092/0 (socket says 172.21.15.53:49092) 2023-07-14T19:17:30.212 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.426+0000 7f1639b09700 1 -- 172.21.15.53:0/990199027 learned_addr learned my addr 172.21.15.53:0/990199027 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:30.212 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.426+0000 7f1639b09700 1 -- 172.21.15.53:0/990199027 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1634079de0 con 0x7f1634106060 2023-07-14T19:17:30.212 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.426+0000 7f1639b09700 1 --2- 172.21.15.53:0/990199027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f1634108440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16300098a0 tx=0x7f1630009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582116f368509de2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:30.212 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.427+0000 7f1638b07700 1 -- 172.21.15.53:0/990199027 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1630004160 con 0x7f1634106060 2023-07-14T19:17:30.212 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.427+0000 7f1638b07700 1 -- 172.21.15.53:0/990199027 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16300042c0 con 0x7f1634106060 2023-07-14T19:17:30.212 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.427+0000 7f1638b07700 1 -- 172.21.15.53:0/990199027 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16300119b0 con 0x7f1634106060 2023-07-14T19:17:30.212 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.427+0000 7f163bd6d700 1 -- 172.21.15.53:0/990199027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 msgr2=0x7f1634108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.427+0000 7f163bd6d700 1 --2- 172.21.15.53:0/990199027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f1634108440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16300098a0 tx=0x7f1630009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.427+0000 7f163bd6d700 1 -- 172.21.15.53:0/990199027 shutdown_connections 2023-07-14T19:17:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.427+0000 7f163bd6d700 1 --2- 172.21.15.53:0/990199027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f1634108440 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.427+0000 7f163bd6d700 1 -- 172.21.15.53:0/990199027 >> 172.21.15.53:0/990199027 conn(0x7f16340ffac0 msgr2=0x7f1634101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.428+0000 7f163bd6d700 1 -- 172.21.15.53:0/990199027 shutdown_connections 2023-07-14T19:17:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.428+0000 7f163bd6d700 1 -- 172.21.15.53:0/990199027 wait complete. 2023-07-14T19:17:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.428+0000 7f163bd6d700 1 Processor -- start 2023-07-14T19:17:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.428+0000 7f163bd6d700 1 -- start start 2023-07-14T19:17:30.214 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.429+0000 7f163bd6d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f16341059c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:30.214 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.429+0000 7f163bd6d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1634105f00 con 0x7f1634106060 2023-07-14T19:17:30.214 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.429+0000 7f1639b09700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f16341059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:30.214 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.429+0000 7f1639b09700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f16341059c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49094/0 (socket says 172.21.15.53:49094) 2023-07-14T19:17:30.214 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.429+0000 7f1639b09700 1 -- 172.21.15.53:0/1168888558 learned_addr learned my addr 172.21.15.53:0/1168888558 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:30.214 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.429+0000 7f1639b09700 1 -- 172.21.15.53:0/1168888558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1630009550 con 0x7f1634106060 2023-07-14T19:17:30.214 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.429+0000 7f1639b09700 1 --2- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f16341059c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1630003e60 tx=0x7f1630011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:30.215 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.429+0000 7f162affd700 1 -- 172.21.15.53:0/1168888558 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1630003970 con 0x7f1634106060 2023-07-14T19:17:30.215 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.430+0000 7f162affd700 1 -- 172.21.15.53:0/1168888558 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1630003ad0 con 0x7f1634106060 2023-07-14T19:17:30.215 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.430+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1634104070 con 0x7f1634106060 2023-07-14T19:17:30.215 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.430+0000 7f162affd700 1 -- 172.21.15.53:0/1168888558 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1630019590 con 0x7f1634106060 2023-07-14T19:17:30.215 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.430+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16341044f0 con 0x7f1634106060 2023-07-14T19:17:30.215 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.431+0000 7f162affd700 1 -- 172.21.15.53:0/1168888558 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f163001a430 con 0x7f1634106060 2023-07-14T19:17:30.215 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.431+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1634061c70 con 0x7f1634106060 2023-07-14T19:17:30.215 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.431+0000 7f162affd700 1 --2- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f16200386a0 0x7f162003ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:30.216 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.431+0000 7f162affd700 1 -- 172.21.15.53:0/1168888558 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f163004c7d0 con 0x7f1634106060 2023-07-14T19:17:30.216 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.431+0000 7f1639308700 1 --2- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f16200386a0 0x7f162003ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:30.216 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.431+0000 7f1639308700 1 --2- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f16200386a0 0x7f162003ab50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1624006fd0 tx=0x7f1624006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:30.216 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.436+0000 7f162affd700 1 -- 172.21.15.53:0/1168888558 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1630003c40 con 0x7f1634106060 2023-07-14T19:17:30.217 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.798+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1634192c80 con 0x7f1634106060 2023-07-14T19:17:30.217 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.802+0000 7f162affd700 1 -- 172.21.15.53:0/1168888558 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1630023b80 con 0x7f1634106060 2023-07-14T19:17:30.217 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f16200386a0 msgr2=0x7f162003ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:30.217 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 --2- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f16200386a0 0x7f162003ab50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1624006fd0 tx=0x7f1624006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:30.217 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 msgr2=0x7f16341059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:30.217 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 --2- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f16341059c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1630003e60 tx=0x7f1630011aa0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.217 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 shutdown_connections 2023-07-14T19:17:30.217 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 --2- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f16200386a0 0x7f162003ab50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.218 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 --2- 172.21.15.53:0/1168888558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634106060 0x7f16341059c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:30.218 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 >> 172.21.15.53:0/1168888558 conn(0x7f16340ffac0 msgr2=0x7f1634101230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:30.218 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 shutdown_connections 2023-07-14T19:17:30.218 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:29.805+0000 7f163bd6d700 1 -- 172.21.15.53:0/1168888558 wait complete. 2023-07-14T19:17:30.436 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:30 smithi053 ceph-mon[96565]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:30.436 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:30 smithi053 ceph-mon[96565]: Saving service mgr spec with placement count:2 2023-07-14T19:17:30.436 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:30 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1168888558' entity='client.admin' 2023-07-14T19:17:30.436 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:30 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:31.556 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.735+0000 7f1cfde85700 1 Processor -- start 2023-07-14T19:17:31.556 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.736+0000 7f1cfde85700 1 -- start start 2023-07-14T19:17:31.556 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.736+0000 7f1cfde85700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf8105920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:31.556 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.736+0000 7f1cfde85700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8105e60 con 0x7f1cf8105510 2023-07-14T19:17:31.556 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.736+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf8105920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:31.557 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.736+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf8105920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49098/0 (socket says 172.21.15.53:49098) 2023-07-14T19:17:31.557 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.736+0000 7f1cf77fe700 1 -- 172.21.15.53:0/3024354798 learned_addr learned my addr 172.21.15.53:0/3024354798 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:31.557 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.736+0000 7f1cf77fe700 1 -- 172.21.15.53:0/3024354798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf8105fa0 con 0x7f1cf8105510 2023-07-14T19:17:31.557 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.737+0000 7f1cf77fe700 1 --2- 172.21.15.53:0/3024354798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf8105920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098a0 tx=0x7f1ce8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768c9759b6188fdd server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:31.557 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.737+0000 7f1cf67fc700 1 -- 172.21.15.53:0/3024354798 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8004160 con 0x7f1cf8105510 2023-07-14T19:17:31.557 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.737+0000 7f1cf67fc700 1 -- 172.21.15.53:0/3024354798 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ce80042c0 con 0x7f1cf8105510 2023-07-14T19:17:31.557 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.737+0000 7f1cf67fc700 1 -- 172.21.15.53:0/3024354798 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ce80119b0 con 0x7f1cf8105510 2023-07-14T19:17:31.558 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.738+0000 7f1cfde85700 1 -- 172.21.15.53:0/3024354798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 msgr2=0x7f1cf8105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:31.558 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.738+0000 7f1cfde85700 1 --2- 172.21.15.53:0/3024354798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf8105920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098a0 tx=0x7f1ce8009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.558 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.738+0000 7f1cfde85700 1 -- 172.21.15.53:0/3024354798 shutdown_connections 2023-07-14T19:17:31.558 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.738+0000 7f1cfde85700 1 --2- 172.21.15.53:0/3024354798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf8105920 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.558 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.738+0000 7f1cfde85700 1 -- 172.21.15.53:0/3024354798 >> 172.21.15.53:0/3024354798 conn(0x7f1cf8100bc0 msgr2=0x7f1cf8102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:31.558 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.738+0000 7f1cfde85700 1 -- 172.21.15.53:0/3024354798 shutdown_connections 2023-07-14T19:17:31.558 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.738+0000 7f1cfde85700 1 -- 172.21.15.53:0/3024354798 wait complete. 2023-07-14T19:17:31.558 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.738+0000 7f1cfde85700 1 Processor -- start 2023-07-14T19:17:31.559 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.739+0000 7f1cfde85700 1 -- start start 2023-07-14T19:17:31.559 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.739+0000 7f1cfde85700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf819b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:31.559 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.739+0000 7f1cfde85700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf819ba30 con 0x7f1cf8105510 2023-07-14T19:17:31.559 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.739+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf819b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:31.559 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.739+0000 7f1cf77fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf819b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49100/0 (socket says 172.21.15.53:49100) 2023-07-14T19:17:31.559 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.739+0000 7f1cf77fe700 1 -- 172.21.15.53:0/3054926888 learned_addr learned my addr 172.21.15.53:0/3054926888 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:31.559 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.739+0000 7f1cf77fe700 1 -- 172.21.15.53:0/3054926888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce8009550 con 0x7f1cf8105510 2023-07-14T19:17:31.560 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.739+0000 7f1cf77fe700 1 --2- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf819b4f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8011980 tx=0x7f1ce8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:31.560 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.740+0000 7f1cf4ff9700 1 -- 172.21.15.53:0/3054926888 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ce8003a60 con 0x7f1cf8105510 2023-07-14T19:17:31.560 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.740+0000 7f1cf4ff9700 1 -- 172.21.15.53:0/3054926888 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ce8003bc0 con 0x7f1cf8105510 2023-07-14T19:17:31.560 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.740+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf819bc30 con 0x7f1cf8105510 2023-07-14T19:17:31.560 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.740+0000 7f1cf4ff9700 1 -- 172.21.15.53:0/3054926888 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ce801a670 con 0x7f1cf8105510 2023-07-14T19:17:31.560 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.740+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf819c050 con 0x7f1cf8105510 2023-07-14T19:17:31.561 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.741+0000 7f1cf4ff9700 1 -- 172.21.15.53:0/3054926888 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f1ce801a8e0 con 0x7f1cf8105510 2023-07-14T19:17:31.561 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.741+0000 7f1cf4ff9700 1 --2- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f1ce0038760 0x7f1ce003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:31.561 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.741+0000 7f1cf4ff9700 1 -- 172.21.15.53:0/3054926888 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1ce804c780 con 0x7f1cf8105510 2023-07-14T19:17:31.561 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.741+0000 7f1cf6ffd700 1 --2- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f1ce0038760 0x7f1ce003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:31.561 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.741+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf8061c70 con 0x7f1cf8105510 2023-07-14T19:17:31.561 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.742+0000 7f1cf6ffd700 1 --2- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f1ce0038760 0x7f1ce003ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1cec006fd0 tx=0x7f1cec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:31.561 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:30.746+0000 7f1cf4ff9700 1 -- 172.21.15.53:0/3054926888 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1ce801ab90 con 0x7f1cf8105510 2023-07-14T19:17:31.562 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.114+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f1cf819f020 con 0x7f1cf8105510 2023-07-14T19:17:31.562 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.118+0000 7f1cf4ff9700 1 -- 172.21.15.53:0/3054926888 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f1ce8018350 con 0x7f1cf8105510 2023-07-14T19:17:31.562 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.120+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f1ce0038760 msgr2=0x7f1ce003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:31.562 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.120+0000 7f1cfde85700 1 --2- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f1ce0038760 0x7f1ce003ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1cec006fd0 tx=0x7f1cec006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:31.562 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.120+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 msgr2=0x7f1cf819b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:31.562 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.120+0000 7f1cfde85700 1 --2- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf819b4f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8011980 tx=0x7f1ce8011a60 comp rx=0 tx=0).stop 2023-07-14T19:17:31.562 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.122+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 shutdown_connections 2023-07-14T19:17:31.562 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.122+0000 7f1cfde85700 1 --2- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f1ce0038760 0x7f1ce003ac10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.563 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.122+0000 7f1cfde85700 1 --2- 172.21.15.53:0/3054926888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf8105510 0x7f1cf819b4f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:31.563 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.122+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 >> 172.21.15.53:0/3054926888 conn(0x7f1cf8100bc0 msgr2=0x7f1cf8102280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:31.563 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.123+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 shutdown_connections 2023-07-14T19:17:31.563 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:31.123+0000 7f1cfde85700 1 -- 172.21.15.53:0/3054926888 wait complete. 2023-07-14T19:17:31.563 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling the dashboard module... 2023-07-14T19:17:32.048 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:31 smithi053 ceph-mon[96565]: from='mgr.14120 172.21.15.53:0/3128988526' entity='mgr.x' 2023-07-14T19:17:32.048 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:31 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3054926888' entity='client.admin' 2023-07-14T19:17:33.516 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:33 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1681816761' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-14T19:17:33.517 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:33 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: ignoring --setuser ceph since I am not root 2023-07-14T19:17:33.517 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:33 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: ignoring --setgroup ceph since I am not root 2023-07-14T19:17:33.517 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:33 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:33.417+0000 7f4e33494200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:17:33.517 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:33 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:33.514+0000 7f4e33494200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:17:33.604 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.050+0000 7f98cbd40700 1 Processor -- start 2023-07-14T19:17:33.605 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.050+0000 7f98cbd40700 1 -- start start 2023-07-14T19:17:33.605 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.050+0000 7f98cbd40700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:33.605 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.050+0000 7f98cbd40700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c4105ee0 con 0x7f98c4105590 2023-07-14T19:17:33.605 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.050+0000 7f98c9adc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:33.605 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.050+0000 7f98c9adc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41059a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49104/0 (socket says 172.21.15.53:49104) 2023-07-14T19:17:33.605 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.050+0000 7f98c9adc700 1 -- 172.21.15.53:0/3813872259 learned_addr learned my addr 172.21.15.53:0/3813872259 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:33.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.051+0000 7f98c9adc700 1 -- 172.21.15.53:0/3813872259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c4106020 con 0x7f98c4105590 2023-07-14T19:17:33.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.051+0000 7f98c9adc700 1 --2- 172.21.15.53:0/3813872259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41059a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f98c00098a0 tx=0x7f98c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c897682df617058b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:33.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.051+0000 7f98c8ada700 1 -- 172.21.15.53:0/3813872259 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98c0004160 con 0x7f98c4105590 2023-07-14T19:17:33.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.051+0000 7f98c8ada700 1 -- 172.21.15.53:0/3813872259 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98c00042c0 con 0x7f98c4105590 2023-07-14T19:17:33.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.051+0000 7f98c8ada700 1 -- 172.21.15.53:0/3813872259 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98c00119b0 con 0x7f98c4105590 2023-07-14T19:17:33.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.051+0000 7f98cbd40700 1 -- 172.21.15.53:0/3813872259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 msgr2=0x7f98c41059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:33.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.051+0000 7f98cbd40700 1 --2- 172.21.15.53:0/3813872259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41059a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f98c00098a0 tx=0x7f98c0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:33.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.052+0000 7f98cbd40700 1 -- 172.21.15.53:0/3813872259 shutdown_connections 2023-07-14T19:17:33.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.052+0000 7f98cbd40700 1 --2- 172.21.15.53:0/3813872259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41059a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:33.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.052+0000 7f98cbd40700 1 -- 172.21.15.53:0/3813872259 >> 172.21.15.53:0/3813872259 conn(0x7f98c4100c20 msgr2=0x7f98c4103030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:33.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.052+0000 7f98cbd40700 1 -- 172.21.15.53:0/3813872259 shutdown_connections 2023-07-14T19:17:33.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.052+0000 7f98cbd40700 1 -- 172.21.15.53:0/3813872259 wait complete. 2023-07-14T19:17:33.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.052+0000 7f98cbd40700 1 Processor -- start 2023-07-14T19:17:33.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.053+0000 7f98cbd40700 1 -- start start 2023-07-14T19:17:33.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.053+0000 7f98cbd40700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:33.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.053+0000 7f98cbd40700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c4199830 con 0x7f98c4105590 2023-07-14T19:17:33.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.053+0000 7f98c9adc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:33.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.053+0000 7f98c9adc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49106/0 (socket says 172.21.15.53:49106) 2023-07-14T19:17:33.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.053+0000 7f98c9adc700 1 -- 172.21.15.53:0/1681816761 learned_addr learned my addr 172.21.15.53:0/1681816761 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:33.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.053+0000 7f98c9adc700 1 -- 172.21.15.53:0/1681816761 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0009550 con 0x7f98c4105590 2023-07-14T19:17:33.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.053+0000 7f98c9adc700 1 --2- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41992f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f98c0011980 tx=0x7f98c0011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:33.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.054+0000 7f98baffd700 1 -- 172.21.15.53:0/1681816761 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98c0003a60 con 0x7f98c4105590 2023-07-14T19:17:33.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.054+0000 7f98baffd700 1 -- 172.21.15.53:0/1681816761 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98c0003bc0 con 0x7f98c4105590 2023-07-14T19:17:33.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.054+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c4199a30 con 0x7f98c4105590 2023-07-14T19:17:33.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.054+0000 7f98baffd700 1 -- 172.21.15.53:0/1681816761 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98c001a670 con 0x7f98c4105590 2023-07-14T19:17:33.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.054+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c4199e50 con 0x7f98c4105590 2023-07-14T19:17:33.609 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.054+0000 7f98baffd700 1 -- 172.21.15.53:0/1681816761 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f98c001a900 con 0x7f98c4105590 2023-07-14T19:17:33.610 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.055+0000 7f98baffd700 1 --2- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f98b003cb70 0x7f98b003f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:33.610 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.055+0000 7f98baffd700 1 -- 172.21.15.53:0/1681816761 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f98c004d710 con 0x7f98c4105590 2023-07-14T19:17:33.610 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.055+0000 7f98c92db700 1 --2- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f98b003cb70 0x7f98b003f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:33.610 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.055+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c4061c70 con 0x7f98c4105590 2023-07-14T19:17:33.610 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.056+0000 7f98c92db700 1 --2- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f98b003cb70 0x7f98b003f020 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f98b4006fd0 tx=0x7f98b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:33.611 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.061+0000 7f98baffd700 1 -- 172.21.15.53:0/1681816761 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f98c0048070 con 0x7f98c4105590 2023-07-14T19:17:33.611 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:32.541+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f98c407a490 con 0x7f98c4105590 2023-07-14T19:17:33.611 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.202+0000 7f98baffd700 1 -- 172.21.15.53:0/1681816761 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f98c002d910 con 0x7f98c4105590 2023-07-14T19:17:33.611 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.204+0000 7f98baffd700 1 -- 172.21.15.53:0/1681816761 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f98c004d060 con 0x7f98c4105590 2023-07-14T19:17:33.611 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f98b003cb70 msgr2=0x7f98b003f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:33.611 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 --2- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f98b003cb70 0x7f98b003f020 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f98b4006fd0 tx=0x7f98b4006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:33.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 msgr2=0x7f98c41992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:33.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 --2- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41992f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f98c0011980 tx=0x7f98c0011a60 comp rx=0 tx=0).stop 2023-07-14T19:17:33.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 shutdown_connections 2023-07-14T19:17:33.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 --2- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f98b003cb70 0x7f98b003f020 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:33.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 --2- 172.21.15.53:0/1681816761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c4105590 0x7f98c41992f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:33.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 >> 172.21.15.53:0/1681816761 conn(0x7f98c4100c20 msgr2=0x7f98c41022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:33.612 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 shutdown_connections 2023-07-14T19:17:33.613 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:33.207+0000 7f98cbd40700 1 -- 172.21.15.53:0/1681816761 wait complete. 2023-07-14T19:17:33.791 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:33 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:33.626+0000 7f4e33494200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:17:33.792 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:33 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:33.741+0000 7f4e33494200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:17:34.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:33 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:33.954+0000 7f4e33494200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:17:34.469 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:34 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:34.357+0000 7f4e33494200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:17:34.469 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:34 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1681816761' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-14T19:17:34.469 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:34 smithi053 ceph-mon[96565]: mgrmap e9: x(active, since 17s) 2023-07-14T19:17:34.740 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:34 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:34.466+0000 7f4e33494200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:17:34.741 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:34 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:34.617+0000 7f4e33494200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:17:34.992 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:34 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:34.739+0000 7f4e33494200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:17:35.052 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:17:35.053 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-14T19:17:35.053 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T19:17:35.053 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T19:17:35.053 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T19:17:35.053 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:17:35.054 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.137+0000 7fdbb367c700 1 Processor -- start 2023-07-14T19:17:35.054 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.138+0000 7fdbb367c700 1 -- start start 2023-07-14T19:17:35.054 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.138+0000 7fdbb367c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac108410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:35.054 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.138+0000 7fdbb367c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac073de0 con 0x7fdbac106030 2023-07-14T19:17:35.054 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.138+0000 7fdbb1418700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:35.055 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.138+0000 7fdbb1418700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac108410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49120/0 (socket says 172.21.15.53:49120) 2023-07-14T19:17:35.055 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.138+0000 7fdbb1418700 1 -- 172.21.15.53:0/3936894193 learned_addr learned my addr 172.21.15.53:0/3936894193 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:35.055 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.138+0000 7fdbb1418700 1 -- 172.21.15.53:0/3936894193 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac073f20 con 0x7fdbac106030 2023-07-14T19:17:35.055 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.139+0000 7fdbb1418700 1 --2- 172.21.15.53:0/3936894193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac108410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdba80098a0 tx=0x7fdba8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f636f2ad4ba967 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:35.055 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.139+0000 7fdba3fff700 1 -- 172.21.15.53:0/3936894193 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdba8004160 con 0x7fdbac106030 2023-07-14T19:17:35.055 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.139+0000 7fdba3fff700 1 -- 172.21.15.53:0/3936894193 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdba80042c0 con 0x7fdbac106030 2023-07-14T19:17:35.055 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.139+0000 7fdba3fff700 1 -- 172.21.15.53:0/3936894193 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdba80119b0 con 0x7fdbac106030 2023-07-14T19:17:35.056 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.139+0000 7fdbb367c700 1 -- 172.21.15.53:0/3936894193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 msgr2=0x7fdbac108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:35.056 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.139+0000 7fdbb367c700 1 --2- 172.21.15.53:0/3936894193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac108410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdba80098a0 tx=0x7fdba8009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:35.056 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.139+0000 7fdbb367c700 1 -- 172.21.15.53:0/3936894193 shutdown_connections 2023-07-14T19:17:35.056 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.139+0000 7fdbb367c700 1 --2- 172.21.15.53:0/3936894193 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac108410 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:35.056 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.140+0000 7fdbb367c700 1 -- 172.21.15.53:0/3936894193 >> 172.21.15.53:0/3936894193 conn(0x7fdbac0ffab0 msgr2=0x7fdbac101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:35.056 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.140+0000 7fdbb367c700 1 -- 172.21.15.53:0/3936894193 shutdown_connections 2023-07-14T19:17:35.056 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.140+0000 7fdbb367c700 1 -- 172.21.15.53:0/3936894193 wait complete. 2023-07-14T19:17:35.057 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.140+0000 7fdbb367c700 1 Processor -- start 2023-07-14T19:17:35.057 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.140+0000 7fdbb367c700 1 -- start start 2023-07-14T19:17:35.057 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdbb367c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac19b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:35.057 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdbb367c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac19ba40 con 0x7fdbac106030 2023-07-14T19:17:35.057 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdbb1418700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac19b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:35.057 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdbb1418700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac19b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49122/0 (socket says 172.21.15.53:49122) 2023-07-14T19:17:35.057 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdbb1418700 1 -- 172.21.15.53:0/3946778124 learned_addr learned my addr 172.21.15.53:0/3946778124 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:35.058 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdbb1418700 1 -- 172.21.15.53:0/3946778124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba8009550 con 0x7fdbac106030 2023-07-14T19:17:35.058 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdbb1418700 1 --2- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac19b500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdba8000c00 tx=0x7fdba8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:35.058 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdba27fc700 1 -- 172.21.15.53:0/3946778124 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdba80039f0 con 0x7fdbac106030 2023-07-14T19:17:35.058 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdba27fc700 1 -- 172.21.15.53:0/3946778124 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdba8003b50 con 0x7fdbac106030 2023-07-14T19:17:35.058 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbac19bc40 con 0x7fdbac106030 2023-07-14T19:17:35.058 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdba27fc700 1 -- 172.21.15.53:0/3946778124 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdba801a780 con 0x7fdbac106030 2023-07-14T19:17:35.058 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.141+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbac19c060 con 0x7fdbac106030 2023-07-14T19:17:35.058 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.142+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbac061c70 con 0x7fdbac106030 2023-07-14T19:17:35.059 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.143+0000 7fdba27fc700 1 -- 172.21.15.53:0/3946778124 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fdba801a9f0 con 0x7fdbac106030 2023-07-14T19:17:35.059 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.143+0000 7fdba27fc700 1 --2- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdb980387b0 0x7fdb9803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:35.059 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.143+0000 7fdba27fc700 1 -- 172.21.15.53:0/3946778124 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdba804ccb0 con 0x7fdbac106030 2023-07-14T19:17:35.059 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.143+0000 7fdbb0c17700 1 -- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdb980387b0 msgr2=0x7fdb9803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2149831 2023-07-14T19:17:35.059 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.143+0000 7fdbb0c17700 1 --2- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdb980387b0 0x7fdb9803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:17:35.059 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.152+0000 7fdba27fc700 1 -- 172.21.15.53:0/3946778124 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdba801aca0 con 0x7fdbac106030 2023-07-14T19:17:35.059 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.344+0000 7fdbb0c17700 1 -- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdb980387b0 msgr2=0x7fdb9803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2149831 2023-07-14T19:17:35.060 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.344+0000 7fdbb0c17700 1 --2- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdb980387b0 0x7fdb9803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:17:35.060 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.623+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fdbac19eb90 con 0x7fdbac106030 2023-07-14T19:17:35.060 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.623+0000 7fdba27fc700 1 -- 172.21.15.53:0/3946778124 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fdba8018350 con 0x7fdbac106030 2023-07-14T19:17:35.060 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdb980387b0 msgr2=0x7fdb9803ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:17:35.060 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 --2- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdb980387b0 0x7fdb9803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:35.060 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 msgr2=0x7fdbac19b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:35.060 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 --2- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac19b500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdba8000c00 tx=0x7fdba8011a30 comp rx=0 tx=0).stop 2023-07-14T19:17:35.060 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 shutdown_connections 2023-07-14T19:17:35.061 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 --2- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7fdb980387b0 0x7fdb9803ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:35.061 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 --2- 172.21.15.53:0/3946778124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbac106030 0x7fdbac19b500 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:35.061 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 >> 172.21.15.53:0/3946778124 conn(0x7fdbac0ffab0 msgr2=0x7fdbac101190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:35.061 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 shutdown_connections 2023-07-14T19:17:35.061 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:34.626+0000 7fdbb367c700 1 -- 172.21.15.53:0/3946778124 wait complete. 2023-07-14T19:17:35.061 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for the mgr to restart... 2023-07-14T19:17:35.061 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr epoch 9... 2023-07-14T19:17:35.251 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:35 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3946778124' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T19:17:35.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:35 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:35.260+0000 7f4e33494200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:17:36.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:35 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:35.949+0000 7f4e33494200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:17:36.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:36 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:36.141+0000 7f4e33494200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:17:37.435 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:37 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:37.103+0000 7f4e33494200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:17:37.435 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:37 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:37.214+0000 7f4e33494200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:17:37.786 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:37 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:37.433+0000 7f4e33494200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:17:38.040 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:37 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:37.784+0000 7f4e33494200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:17:38.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:38 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:38.038+0000 7f4e33494200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:17:38.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:38 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:38.221+0000 7f4e33494200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:17:38.822 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:38 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:38.548+0000 7f4e33494200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:17:39.779 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:39 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:39.375+0000 7f4e33494200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:17:39.780 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:39 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:39.533+0000 7f4e33494200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:17:40.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:39 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:39.777+0000 7f4e33494200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:17:40.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:39 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:39.915+0000 7f4e33494200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:17:40.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:40 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:40.187+0000 7f4e33494200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:17:41.572 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:17:41.226+0000 7f4e33494200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:17:41.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: Active manager daemon x restarted 2023-07-14T19:17:41.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: Activating manager daemon x 2023-07-14T19:17:41.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: osdmap e3: 0 total, 0 up, 0 in 2023-07-14T19:17:41.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: mgrmap e10: x(active, starting, since 0.00316759s) 2023-07-14T19:17:41.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:17:41.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T19:17:41.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T19:17:41.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T19:17:41.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T19:17:41.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: Manager daemon x is now available 2023-07-14T19:17:41.576 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:41.576 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:42.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T19:17:42.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T19:17:42.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:42 smithi053 ceph-mon[96565]: [14/Jul/2023:19:17:41] ENGINE Bus STARTING 2023-07-14T19:17:42.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:42 smithi053 ceph-mon[96565]: [14/Jul/2023:19:17:42] ENGINE Serving on http://172.21.15.53:8765 2023-07-14T19:17:42.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:42 smithi053 ceph-mon[96565]: [14/Jul/2023:19:17:42] ENGINE Serving on https://172.21.15.53:7150 2023-07-14T19:17:42.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:42 smithi053 ceph-mon[96565]: [14/Jul/2023:19:17:42] ENGINE Bus STARTED 2023-07-14T19:17:42.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:42 smithi053 ceph-mon[96565]: mgrmap e11: x(active, since 1.00539s) 2023-07-14T19:17:42.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:42 smithi053 ceph-mon[96565]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T19:17:42.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:42 smithi053 ceph-mon[96565]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T19:17:42.718 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:17:42.719 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-14T19:17:42.719 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T19:17:42.719 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:17:42.719 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.501+0000 7f4d995d5700 1 Processor -- start 2023-07-14T19:17:42.720 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.501+0000 7f4d995d5700 1 -- start start 2023-07-14T19:17:42.720 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.502+0000 7f4d995d5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d94108450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:42.720 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.502+0000 7f4d995d5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d94073da0 con 0x7f4d94106070 2023-07-14T19:17:42.720 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.502+0000 7f4d92ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d94108450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:42.720 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.502+0000 7f4d92ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d94108450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49130/0 (socket says 172.21.15.53:49130) 2023-07-14T19:17:42.721 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.502+0000 7f4d92ffd700 1 -- 172.21.15.53:0/2538125140 learned_addr learned my addr 172.21.15.53:0/2538125140 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:42.721 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.502+0000 7f4d92ffd700 1 -- 172.21.15.53:0/2538125140 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d94073ee0 con 0x7f4d94106070 2023-07-14T19:17:42.721 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.502+0000 7f4d92ffd700 1 --2- 172.21.15.53:0/2538125140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d94108450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4d840098a0 tx=0x7f4d84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6c2d6a86860483 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:42.721 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.502+0000 7f4d91ffb700 1 -- 172.21.15.53:0/2538125140 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d84004160 con 0x7f4d94106070 2023-07-14T19:17:42.721 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.502+0000 7f4d91ffb700 1 -- 172.21.15.53:0/2538125140 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d840042c0 con 0x7f4d94106070 2023-07-14T19:17:42.721 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.503+0000 7f4d91ffb700 1 -- 172.21.15.53:0/2538125140 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d840119b0 con 0x7f4d94106070 2023-07-14T19:17:42.722 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.503+0000 7f4d995d5700 1 -- 172.21.15.53:0/2538125140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 msgr2=0x7f4d94108450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:42.722 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.503+0000 7f4d995d5700 1 --2- 172.21.15.53:0/2538125140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d94108450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4d840098a0 tx=0x7f4d84009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:42.722 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.503+0000 7f4d995d5700 1 -- 172.21.15.53:0/2538125140 shutdown_connections 2023-07-14T19:17:42.722 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.503+0000 7f4d995d5700 1 --2- 172.21.15.53:0/2538125140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d94108450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:42.722 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.503+0000 7f4d995d5700 1 -- 172.21.15.53:0/2538125140 >> 172.21.15.53:0/2538125140 conn(0x7f4d940ffab0 msgr2=0x7f4d94101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:42.722 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.503+0000 7f4d995d5700 1 -- 172.21.15.53:0/2538125140 shutdown_connections 2023-07-14T19:17:42.722 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.503+0000 7f4d995d5700 1 -- 172.21.15.53:0/2538125140 wait complete. 2023-07-14T19:17:42.723 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.504+0000 7f4d995d5700 1 Processor -- start 2023-07-14T19:17:42.723 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.504+0000 7f4d995d5700 1 -- start start 2023-07-14T19:17:42.723 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d995d5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d941992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:42.723 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d995d5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d94199800 con 0x7f4d94106070 2023-07-14T19:17:42.723 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d92ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d941992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:42.723 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d92ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d941992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49132/0 (socket says 172.21.15.53:49132) 2023-07-14T19:17:42.723 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d92ffd700 1 -- 172.21.15.53:0/4014409883 learned_addr learned my addr 172.21.15.53:0/4014409883 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:42.724 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d92ffd700 1 -- 172.21.15.53:0/4014409883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d84009550 con 0x7f4d94106070 2023-07-14T19:17:42.724 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d92ffd700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d941992c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4d84003e60 tx=0x7f4d84011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:42.724 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d84003970 con 0x7f4d94106070 2023-07-14T19:17:42.724 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d84003ad0 con 0x7f4d94106070 2023-07-14T19:17:42.724 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d84019590 con 0x7f4d94106070 2023-07-14T19:17:42.724 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.505+0000 7f4d995d5700 1 -- 172.21.15.53:0/4014409883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d94199a60 con 0x7f4d94106070 2023-07-14T19:17:42.725 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.506+0000 7f4d995d5700 1 -- 172.21.15.53:0/4014409883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d94199ee0 con 0x7f4d94106070 2023-07-14T19:17:42.725 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.507+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f4d84019850 con 0x7f4d94106070 2023-07-14T19:17:42.725 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.507+0000 7f4d7bfff700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 0x7f4d7c03aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:42.725 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.507+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 --> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c03b2b0 con 0x7f4d7c0386f0 2023-07-14T19:17:42.725 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.507+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4d8404c8c0 con 0x7f4d94106070 2023-07-14T19:17:42.725 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.507+0000 7f4d927fc700 1 -- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 msgr2=0x7f4d7c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2149831 2023-07-14T19:17:42.725 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.507+0000 7f4d927fc700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 0x7f4d7c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:17:42.726 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.708+0000 7f4d927fc700 1 -- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 msgr2=0x7f4d7c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2149831 2023-07-14T19:17:42.726 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:35.708+0000 7f4d927fc700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 0x7f4d7c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:17:42.726 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:36.109+0000 7f4d927fc700 1 -- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 msgr2=0x7f4d7c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2149831 2023-07-14T19:17:42.726 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:36.109+0000 7f4d927fc700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 0x7f4d7c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T19:17:42.726 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:36.910+0000 7f4d927fc700 1 -- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 msgr2=0x7f4d7c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2149831 2023-07-14T19:17:42.726 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:36.910+0000 7f4d927fc700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 0x7f4d7c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T19:17:42.726 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:38.511+0000 7f4d927fc700 1 -- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 msgr2=0x7f4d7c03aba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.53:6800/2149831 2023-07-14T19:17:42.727 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:38.511+0000 7f4d927fc700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 0x7f4d7c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T19:17:42.727 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:41.241+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f4d84019b50 con 0x7f4d94106070 2023-07-14T19:17:42.727 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:41.241+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 msgr2=0x7f4d7c03aba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:17:42.727 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:41.241+0000 7f4d7bfff700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 0x7f4d7c03aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:42.727 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.243+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f4d8404e1c0 con 0x7f4d94106070 2023-07-14T19:17:42.727 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.243+0000 7f4d7bfff700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d7c03bff0 0x7f4d7c03e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:42.728 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.243+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7c03b2b0 con 0x7f4d7c03bff0 2023-07-14T19:17:42.728 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.243+0000 7f4d927fc700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d7c03bff0 0x7f4d7c03e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:42.728 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.244+0000 7f4d927fc700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d7c03bff0 0x7f4d7c03e3d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4d88003a10 tx=0x7f4d880092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:42.728 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.245+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f4d7c03b2b0 con 0x7f4d7c03bff0 2023-07-14T19:17:42.729 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.255+0000 7f4d995d5700 1 -- 172.21.15.53:0/4014409883 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4d9419a600 con 0x7f4d7c03bff0 2023-07-14T19:17:42.729 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d7bfff700 1 -- 172.21.15.53:0/4014409883 <== mgr.14152 v2:172.21.15.53:6800/3014344459 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f4d9419a600 con 0x7f4d7c03bff0 2023-07-14T19:17:42.729 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 -- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d7c03bff0 msgr2=0x7f4d7c03e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:42.729 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d7c03bff0 0x7f4d7c03e3d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4d88003a10 tx=0x7f4d880092b0 comp rx=0 tx=0).stop 2023-07-14T19:17:42.729 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 -- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 msgr2=0x7f4d941992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:42.729 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d941992c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4d84003e60 tx=0x7f4d84011aa0 comp rx=0 tx=0).stop 2023-07-14T19:17:42.730 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 -- 172.21.15.53:0/4014409883 shutdown_connections 2023-07-14T19:17:42.730 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d94106070 0x7f4d941992c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:42.730 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d7c03bff0 0x7f4d7c03e3d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:42.730 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 --2- 172.21.15.53:0/4014409883 >> [v2:172.21.15.53:6800/2149831,v1:172.21.15.53:6801/2149831] conn(0x7f4d7c0386f0 0x7f4d7c03aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:42.731 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 -- 172.21.15.53:0/4014409883 >> 172.21.15.53:0/4014409883 conn(0x7f4d940ffab0 msgr2=0x7f4d94101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:42.731 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 -- 172.21.15.53:0/4014409883 shutdown_connections 2023-07-14T19:17:42.731 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:42.256+0000 7f4d995d5700 1 -- 172.21.15.53:0/4014409883 wait complete. 2023-07-14T19:17:42.731 INFO:teuthology.orchestra.run.smithi053.stdout:mgr epoch 9 is available 2023-07-14T19:17:42.732 INFO:teuthology.orchestra.run.smithi053.stdout:Generating a dashboard self-signed certificate... 2023-07-14T19:17:44.368 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-14T19:17:44.369 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.383+0000 7f225189b700 1 Processor -- start 2023-07-14T19:17:44.369 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.383+0000 7f225189b700 1 -- start start 2023-07-14T19:17:44.369 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.383+0000 7f225189b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c073d60 0x7f224c10a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:44.369 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.383+0000 7f225189b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c105eb0 con 0x7f224c073d60 2023-07-14T19:17:44.369 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.384+0000 7f224affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c073d60 0x7f224c10a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:44.369 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.384+0000 7f224affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c073d60 0x7f224c10a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49186/0 (socket says 172.21.15.53:49186) 2023-07-14T19:17:44.369 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.384+0000 7f224affd700 1 -- 172.21.15.53:0/3416696710 learned_addr learned my addr 172.21.15.53:0/3416696710 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:44.370 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.384+0000 7f224affd700 1 -- 172.21.15.53:0/3416696710 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c10aa00 con 0x7f224c073d60 2023-07-14T19:17:44.370 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.384+0000 7f224affd700 1 --2- 172.21.15.53:0/3416696710 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c073d60 0x7f224c10a4c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f223c0098a0 tx=0x7f223c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e5817c199db242 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:44.370 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.384+0000 7f2249ffb700 1 -- 172.21.15.53:0/3416696710 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f223c004160 con 0x7f224c073d60 2023-07-14T19:17:44.370 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.384+0000 7f2249ffb700 1 -- 172.21.15.53:0/3416696710 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f223c0042c0 con 0x7f224c073d60 2023-07-14T19:17:44.370 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.384+0000 7f2249ffb700 1 -- 172.21.15.53:0/3416696710 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f223c0119b0 con 0x7f224c073d60 2023-07-14T19:17:44.370 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.385+0000 7f225189b700 1 -- 172.21.15.53:0/3416696710 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c073d60 msgr2=0x7f224c10a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:44.370 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.385+0000 7f225189b700 1 --2- 172.21.15.53:0/3416696710 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c073d60 0x7f224c10a4c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f223c0098a0 tx=0x7f223c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:44.371 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.385+0000 7f225189b700 1 -- 172.21.15.53:0/3416696710 shutdown_connections 2023-07-14T19:17:44.371 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.385+0000 7f225189b700 1 --2- 172.21.15.53:0/3416696710 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c073d60 0x7f224c10a4c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:44.371 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.385+0000 7f225189b700 1 -- 172.21.15.53:0/3416696710 >> 172.21.15.53:0/3416696710 conn(0x7f224c0ffab0 msgr2=0x7f224c101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:44.371 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.385+0000 7f225189b700 1 -- 172.21.15.53:0/3416696710 shutdown_connections 2023-07-14T19:17:44.371 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.385+0000 7f225189b700 1 -- 172.21.15.53:0/3416696710 wait complete. 2023-07-14T19:17:44.371 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.386+0000 7f225189b700 1 Processor -- start 2023-07-14T19:17:44.371 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.386+0000 7f225189b700 1 -- start start 2023-07-14T19:17:44.371 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.386+0000 7f225189b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c19b4e0 0x7f224c19b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:44.372 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.386+0000 7f225189b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c19be30 con 0x7f224c19b4e0 2023-07-14T19:17:44.372 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.386+0000 7f224affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c19b4e0 0x7f224c19b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:44.372 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.386+0000 7f224affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c19b4e0 0x7f224c19b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49188/0 (socket says 172.21.15.53:49188) 2023-07-14T19:17:44.372 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.386+0000 7f224affd700 1 -- 172.21.15.53:0/815697436 learned_addr learned my addr 172.21.15.53:0/815697436 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:44.372 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.386+0000 7f224affd700 1 -- 172.21.15.53:0/815697436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c009550 con 0x7f224c19b4e0 2023-07-14T19:17:44.372 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.387+0000 7f224affd700 1 --2- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c19b4e0 0x7f224c19b8f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f223c000c00 tx=0x7f223c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:44.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.387+0000 7f2250899700 1 -- 172.21.15.53:0/815697436 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f223c0039f0 con 0x7f224c19b4e0 2023-07-14T19:17:44.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.387+0000 7f2250899700 1 -- 172.21.15.53:0/815697436 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f223c003b50 con 0x7f224c19b4e0 2023-07-14T19:17:44.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.387+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c19c030 con 0x7f224c19b4e0 2023-07-14T19:17:44.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.387+0000 7f2250899700 1 -- 172.21.15.53:0/815697436 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f223c01a780 con 0x7f224c19b4e0 2023-07-14T19:17:44.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.387+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c19ec90 con 0x7f224c19b4e0 2023-07-14T19:17:44.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.388+0000 7f2250899700 1 -- 172.21.15.53:0/815697436 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f223c01aa40 con 0x7f224c19b4e0 2023-07-14T19:17:44.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.388+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c195510 con 0x7f224c19b4e0 2023-07-14T19:17:44.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.388+0000 7f2250899700 1 --2- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2234038760 0x7f223403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:44.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.388+0000 7f2250899700 1 -- 172.21.15.53:0/815697436 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f223c04c8b0 con 0x7f224c19b4e0 2023-07-14T19:17:44.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.388+0000 7f224a7fc700 1 --2- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2234038760 0x7f223403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:44.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.389+0000 7f224a7fc700 1 --2- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2234038760 0x7f223403ac10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2240006fd0 tx=0x7f2240006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:44.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.393+0000 7f2250899700 1 -- 172.21.15.53:0/815697436 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f223c0237b0 con 0x7f224c19b4e0 2023-07-14T19:17:44.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.762+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f224c02caa0 con 0x7f2234038760 2023-07-14T19:17:44.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.958+0000 7f2250899700 1 -- 172.21.15.53:0/815697436 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f224c02caa0 con 0x7f2234038760 2023-07-14T19:17:44.375 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2234038760 msgr2=0x7f223403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:44.375 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 --2- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2234038760 0x7f223403ac10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2240006fd0 tx=0x7f2240006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:44.375 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c19b4e0 msgr2=0x7f224c19b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:44.375 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 --2- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c19b4e0 0x7f224c19b8f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f223c000c00 tx=0x7f223c011a30 comp rx=0 tx=0).stop 2023-07-14T19:17:44.375 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 shutdown_connections 2023-07-14T19:17:44.375 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 --2- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2234038760 0x7f223403ac10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:44.375 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 --2- 172.21.15.53:0/815697436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224c19b4e0 0x7f224c19b8f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:44.375 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 >> 172.21.15.53:0/815697436 conn(0x7f224c0ffab0 msgr2=0x7f224c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:44.376 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 shutdown_connections 2023-07-14T19:17:44.376 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:43.961+0000 7f225189b700 1 -- 172.21.15.53:0/815697436 wait complete. 2023-07-14T19:17:44.376 INFO:teuthology.orchestra.run.smithi053.stdout:Creating initial admin user... 2023-07-14T19:17:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:44 smithi053 ceph-mon[96565]: mgrmap e12: x(active, since 2s) 2023-07-14T19:17:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:44.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:45.552 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:45 smithi053 ceph-mon[96565]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:46.014 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$FBIIJiwJV6f5ruU4pMV91.RwvTfBNUf0Q4LfMH9CfrX6LgWYv2mbS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689362265, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-14T19:17:46.015 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.863+0000 7f79d33bd700 1 Processor -- start 2023-07-14T19:17:46.015 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.863+0000 7f79d33bd700 1 -- start start 2023-07-14T19:17:46.015 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.863+0000 7f79d33bd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc108440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:46.015 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.863+0000 7f79d33bd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc079ca0 con 0x7f79cc106060 2023-07-14T19:17:46.015 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.864+0000 7f79d1159700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:46.015 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.864+0000 7f79d1159700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49192/0 (socket says 172.21.15.53:49192) 2023-07-14T19:17:46.016 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.864+0000 7f79d1159700 1 -- 172.21.15.53:0/3489526956 learned_addr learned my addr 172.21.15.53:0/3489526956 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:46.016 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.864+0000 7f79d1159700 1 -- 172.21.15.53:0/3489526956 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79cc079de0 con 0x7f79cc106060 2023-07-14T19:17:46.016 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.864+0000 7f79d1159700 1 --2- 172.21.15.53:0/3489526956 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc108440 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79c80098a0 tx=0x7f79c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a15008cc961571 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:46.016 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.864+0000 7f79c3fff700 1 -- 172.21.15.53:0/3489526956 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79c8004160 con 0x7f79cc106060 2023-07-14T19:17:46.016 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.864+0000 7f79c3fff700 1 -- 172.21.15.53:0/3489526956 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79c80042c0 con 0x7f79cc106060 2023-07-14T19:17:46.016 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.864+0000 7f79c3fff700 1 -- 172.21.15.53:0/3489526956 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79c80119b0 con 0x7f79cc106060 2023-07-14T19:17:46.016 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.865+0000 7f79d33bd700 1 -- 172.21.15.53:0/3489526956 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 msgr2=0x7f79cc108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:46.017 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.865+0000 7f79d33bd700 1 --2- 172.21.15.53:0/3489526956 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc108440 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79c80098a0 tx=0x7f79c8009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:46.017 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.865+0000 7f79d33bd700 1 -- 172.21.15.53:0/3489526956 shutdown_connections 2023-07-14T19:17:46.017 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.865+0000 7f79d33bd700 1 --2- 172.21.15.53:0/3489526956 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc108440 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:46.017 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.865+0000 7f79d33bd700 1 -- 172.21.15.53:0/3489526956 >> 172.21.15.53:0/3489526956 conn(0x7f79cc0ffac0 msgr2=0x7f79cc101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:46.017 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.865+0000 7f79d33bd700 1 -- 172.21.15.53:0/3489526956 shutdown_connections 2023-07-14T19:17:46.017 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.865+0000 7f79d33bd700 1 -- 172.21.15.53:0/3489526956 wait complete. 2023-07-14T19:17:46.017 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.866+0000 7f79d33bd700 1 Processor -- start 2023-07-14T19:17:46.018 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.866+0000 7f79d33bd700 1 -- start start 2023-07-14T19:17:46.018 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.866+0000 7f79d33bd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc1992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:46.018 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.866+0000 7f79d33bd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79cc199800 con 0x7f79cc106060 2023-07-14T19:17:46.018 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.866+0000 7f79d1159700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc1992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:46.018 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.866+0000 7f79d1159700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc1992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49194/0 (socket says 172.21.15.53:49194) 2023-07-14T19:17:46.018 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.866+0000 7f79d1159700 1 -- 172.21.15.53:0/1877356511 learned_addr learned my addr 172.21.15.53:0/1877356511 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:46.018 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.866+0000 7f79d1159700 1 -- 172.21.15.53:0/1877356511 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c8009550 con 0x7f79cc106060 2023-07-14T19:17:46.018 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.866+0000 7f79d1159700 1 --2- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc1992c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79c8000c00 tx=0x7f79c8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:46.019 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.867+0000 7f79c27fc700 1 -- 172.21.15.53:0/1877356511 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79c80039f0 con 0x7f79cc106060 2023-07-14T19:17:46.019 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.867+0000 7f79c27fc700 1 -- 172.21.15.53:0/1877356511 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79c8003b50 con 0x7f79cc106060 2023-07-14T19:17:46.019 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.867+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79cc199a60 con 0x7f79cc106060 2023-07-14T19:17:46.019 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.867+0000 7f79c27fc700 1 -- 172.21.15.53:0/1877356511 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79c801a780 con 0x7f79cc106060 2023-07-14T19:17:46.019 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.867+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79cc199ee0 con 0x7f79cc106060 2023-07-14T19:17:46.019 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.867+0000 7f79c27fc700 1 -- 172.21.15.53:0/1877356511 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f79c801aa40 con 0x7f79cc106060 2023-07-14T19:17:46.019 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.868+0000 7f79c27fc700 1 --2- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f79b8038760 0x7f79b803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:46.020 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.868+0000 7f79c27fc700 1 -- 172.21.15.53:0/1877356511 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f79c804ca50 con 0x7f79cc106060 2023-07-14T19:17:46.020 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.868+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79cc1932a0 con 0x7f79cc106060 2023-07-14T19:17:46.020 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.868+0000 7f79d0958700 1 --2- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f79b8038760 0x7f79b803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:46.020 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.868+0000 7f79d0958700 1 --2- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f79b8038760 0x7f79b803ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f79bc006fd0 tx=0x7f79bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:46.020 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:44.873+0000 7f79c27fc700 1 -- 172.21.15.53:0/1877356511 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f79c801ad40 con 0x7f79cc106060 2023-07-14T19:17:46.020 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.273+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f79cc061a90 con 0x7f79b8038760 2023-07-14T19:17:46.020 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.521+0000 7f79c27fc700 1 -- 172.21.15.53:0/1877356511 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f79cc061a90 con 0x7f79b8038760 2023-07-14T19:17:46.021 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f79b8038760 msgr2=0x7f79b803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:46.021 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 --2- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f79b8038760 0x7f79b803ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f79bc006fd0 tx=0x7f79bc006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:46.021 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 msgr2=0x7f79cc1992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:46.021 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 --2- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc1992c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79c8000c00 tx=0x7f79c8011a30 comp rx=0 tx=0).stop 2023-07-14T19:17:46.021 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 shutdown_connections 2023-07-14T19:17:46.021 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 --2- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f79b8038760 0x7f79b803ac10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:46.021 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 --2- 172.21.15.53:0/1877356511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79cc106060 0x7f79cc1992c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:46.021 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 >> 172.21.15.53:0/1877356511 conn(0x7f79cc0ffac0 msgr2=0x7f79cc101230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:46.022 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 shutdown_connections 2023-07-14T19:17:46.022 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:45.524+0000 7f79d33bd700 1 -- 172.21.15.53:0/1877356511 wait complete. 2023-07-14T19:17:46.022 INFO:teuthology.orchestra.run.smithi053.stdout:Fetching dashboard port number... 2023-07-14T19:17:46.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:46 smithi053 ceph-mon[96565]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:46.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:47.163 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 8443 2023-07-14T19:17:47.163 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.462+0000 7fe45504f700 1 Processor -- start 2023-07-14T19:17:47.163 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.462+0000 7fe45504f700 1 -- start start 2023-07-14T19:17:47.163 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.462+0000 7fe45504f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe450108450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:47.164 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.463+0000 7fe45504f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe450073da0 con 0x7fe450106070 2023-07-14T19:17:47.164 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.463+0000 7fe44ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe450108450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:47.164 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.463+0000 7fe44ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe450108450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49198/0 (socket says 172.21.15.53:49198) 2023-07-14T19:17:47.164 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.463+0000 7fe44ed9d700 1 -- 172.21.15.53:0/1313576283 learned_addr learned my addr 172.21.15.53:0/1313576283 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:47.164 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.463+0000 7fe44ed9d700 1 -- 172.21.15.53:0/1313576283 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe450073ee0 con 0x7fe450106070 2023-07-14T19:17:47.164 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.463+0000 7fe44ed9d700 1 --2- 172.21.15.53:0/1313576283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe450108450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4400098a0 tx=0x7fe440009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fb28ed9e8cf05 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:47.164 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.463+0000 7fe44dd9b700 1 -- 172.21.15.53:0/1313576283 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe440004160 con 0x7fe450106070 2023-07-14T19:17:47.165 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.463+0000 7fe44dd9b700 1 -- 172.21.15.53:0/1313576283 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe4400042c0 con 0x7fe450106070 2023-07-14T19:17:47.165 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.463+0000 7fe44dd9b700 1 -- 172.21.15.53:0/1313576283 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4400119b0 con 0x7fe450106070 2023-07-14T19:17:47.165 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.464+0000 7fe45504f700 1 -- 172.21.15.53:0/1313576283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 msgr2=0x7fe450108450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:47.165 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.464+0000 7fe45504f700 1 --2- 172.21.15.53:0/1313576283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe450108450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4400098a0 tx=0x7fe440009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:47.165 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.464+0000 7fe45504f700 1 -- 172.21.15.53:0/1313576283 shutdown_connections 2023-07-14T19:17:47.165 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.464+0000 7fe45504f700 1 --2- 172.21.15.53:0/1313576283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe450108450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:47.165 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.464+0000 7fe45504f700 1 -- 172.21.15.53:0/1313576283 >> 172.21.15.53:0/1313576283 conn(0x7fe4500ffab0 msgr2=0x7fe450101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:47.165 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.464+0000 7fe45504f700 1 -- 172.21.15.53:0/1313576283 shutdown_connections 2023-07-14T19:17:47.166 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.464+0000 7fe45504f700 1 -- 172.21.15.53:0/1313576283 wait complete. 2023-07-14T19:17:47.166 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.465+0000 7fe45504f700 1 Processor -- start 2023-07-14T19:17:47.166 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.465+0000 7fe45504f700 1 -- start start 2023-07-14T19:17:47.166 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.465+0000 7fe45504f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe4501059d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:47.166 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.465+0000 7fe45504f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe450105f10 con 0x7fe450106070 2023-07-14T19:17:47.166 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.465+0000 7fe44ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe4501059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:47.166 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.465+0000 7fe44ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe4501059d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49200/0 (socket says 172.21.15.53:49200) 2023-07-14T19:17:47.167 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.465+0000 7fe44ed9d700 1 -- 172.21.15.53:0/2073383946 learned_addr learned my addr 172.21.15.53:0/2073383946 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:47.167 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.466+0000 7fe44ed9d700 1 -- 172.21.15.53:0/2073383946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe440009550 con 0x7fe450106070 2023-07-14T19:17:47.167 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.466+0000 7fe44ed9d700 1 --2- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe4501059d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe440003e60 tx=0x7fe440011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:47.167 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.466+0000 7fe437fff700 1 -- 172.21.15.53:0/2073383946 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe440003970 con 0x7fe450106070 2023-07-14T19:17:47.167 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.466+0000 7fe437fff700 1 -- 172.21.15.53:0/2073383946 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe440003ad0 con 0x7fe450106070 2023-07-14T19:17:47.167 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.466+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe450104080 con 0x7fe450106070 2023-07-14T19:17:47.167 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.466+0000 7fe437fff700 1 -- 172.21.15.53:0/2073383946 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe440019590 con 0x7fe450106070 2023-07-14T19:17:47.168 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.466+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe450104500 con 0x7fe450106070 2023-07-14T19:17:47.168 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.467+0000 7fe437fff700 1 -- 172.21.15.53:0/2073383946 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fe440019840 con 0x7fe450106070 2023-07-14T19:17:47.168 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.467+0000 7fe437fff700 1 --2- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe438038740 0x7fe43803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:47.168 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.467+0000 7fe437fff700 1 -- 172.21.15.53:0/2073383946 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe44004cc30 con 0x7fe450106070 2023-07-14T19:17:47.168 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.467+0000 7fe44e59c700 1 --2- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe438038740 0x7fe43803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:47.168 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.468+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4501095e0 con 0x7fe450106070 2023-07-14T19:17:47.169 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.468+0000 7fe44e59c700 1 --2- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe438038740 0x7fe43803abf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe444006fd0 tx=0x7fe444006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:47.169 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.472+0000 7fe437fff700 1 -- 172.21.15.53:0/2073383946 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe440019b90 con 0x7fe450106070 2023-07-14T19:17:47.169 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.840+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fe4501097a0 con 0x7fe450106070 2023-07-14T19:17:47.170 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.840+0000 7fe437fff700 1 -- 172.21.15.53:0/2073383946 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fe440018350 con 0x7fe450106070 2023-07-14T19:17:47.170 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.842+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe438038740 msgr2=0x7fe43803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:47.170 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.842+0000 7fe45504f700 1 --2- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe438038740 0x7fe43803abf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe444006fd0 tx=0x7fe444006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:47.170 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.842+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 msgr2=0x7fe4501059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:47.171 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.842+0000 7fe45504f700 1 --2- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe4501059d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe440003e60 tx=0x7fe440011aa0 comp rx=0 tx=0).stop 2023-07-14T19:17:47.171 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.842+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 shutdown_connections 2023-07-14T19:17:47.171 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.842+0000 7fe45504f700 1 --2- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe438038740 0x7fe43803abf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:47.171 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.842+0000 7fe45504f700 1 --2- 172.21.15.53:0/2073383946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe450106070 0x7fe4501059d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:47.171 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.842+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 >> 172.21.15.53:0/2073383946 conn(0x7fe4500ffab0 msgr2=0x7fe450101190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:47.171 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.843+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 shutdown_connections 2023-07-14T19:17:47.171 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:46.843+0000 7fe45504f700 1 -- 172.21.15.53:0/2073383946 wait complete. 2023-07-14T19:17:47.174 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:17:47.174 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2023-07-14T19:17:47.181 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:17:47.181 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2023-07-14T19:17:47.181 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2023-07-14T19:17:47.182 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-14T19:17:47.183 INFO:teuthology.orchestra.run.smithi053.stdout:Ceph Dashboard is now available at: 2023-07-14T19:17:47.183 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:47.183 INFO:teuthology.orchestra.run.smithi053.stdout: URL: https://smithi053:8443/ 2023-07-14T19:17:47.183 INFO:teuthology.orchestra.run.smithi053.stdout: User: admin 2023-07-14T19:17:47.183 INFO:teuthology.orchestra.run.smithi053.stdout: Password: 54ox1aiyl1 2023-07-14T19:17:47.184 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:47.184 INFO:teuthology.orchestra.run.smithi053.stdout:Saving cluster configuration to /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config directory 2023-07-14T19:17:47.184 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling autotune for osd_memory_target 2023-07-14T19:17:47.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:47 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2073383946' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-14T19:17:48.533 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.660+0000 7f586bf0e700 1 Processor -- start 2023-07-14T19:17:48.534 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.660+0000 7f586bf0e700 1 -- start start 2023-07-14T19:17:48.534 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.660+0000 7f586bf0e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f5864108440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:48.534 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.660+0000 7f586bf0e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5864079ca0 con 0x7f5864106060 2023-07-14T19:17:48.534 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.661+0000 7f5869caa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f5864108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:48.534 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.661+0000 7f5869caa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f5864108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49208/0 (socket says 172.21.15.53:49208) 2023-07-14T19:17:48.535 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.661+0000 7f5869caa700 1 -- 172.21.15.53:0/2021309072 learned_addr learned my addr 172.21.15.53:0/2021309072 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:48.535 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.661+0000 7f5869caa700 1 -- 172.21.15.53:0/2021309072 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5864079de0 con 0x7f5864106060 2023-07-14T19:17:48.535 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.661+0000 7f5869caa700 1 --2- 172.21.15.53:0/2021309072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f5864108440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58540098a0 tx=0x7f5854009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c09ee170fb6f2d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:48.535 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.661+0000 7f5868ca8700 1 -- 172.21.15.53:0/2021309072 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5854004160 con 0x7f5864106060 2023-07-14T19:17:48.535 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.661+0000 7f5868ca8700 1 -- 172.21.15.53:0/2021309072 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58540042c0 con 0x7f5864106060 2023-07-14T19:17:48.535 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.661+0000 7f5868ca8700 1 -- 172.21.15.53:0/2021309072 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58540119b0 con 0x7f5864106060 2023-07-14T19:17:48.535 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.662+0000 7f586bf0e700 1 -- 172.21.15.53:0/2021309072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 msgr2=0x7f5864108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:48.536 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.662+0000 7f586bf0e700 1 --2- 172.21.15.53:0/2021309072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f5864108440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58540098a0 tx=0x7f5854009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:48.536 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.662+0000 7f586bf0e700 1 -- 172.21.15.53:0/2021309072 shutdown_connections 2023-07-14T19:17:48.536 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.662+0000 7f586bf0e700 1 --2- 172.21.15.53:0/2021309072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f5864108440 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:48.536 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.662+0000 7f586bf0e700 1 -- 172.21.15.53:0/2021309072 >> 172.21.15.53:0/2021309072 conn(0x7f58640ffac0 msgr2=0x7f5864101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:48.536 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.662+0000 7f586bf0e700 1 -- 172.21.15.53:0/2021309072 shutdown_connections 2023-07-14T19:17:48.536 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.662+0000 7f586bf0e700 1 -- 172.21.15.53:0/2021309072 wait complete. 2023-07-14T19:17:48.536 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.663+0000 7f586bf0e700 1 Processor -- start 2023-07-14T19:17:48.536 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.663+0000 7f586bf0e700 1 -- start start 2023-07-14T19:17:48.537 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.663+0000 7f586bf0e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f58641992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:48.537 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.663+0000 7f586bf0e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5864199800 con 0x7f5864106060 2023-07-14T19:17:48.537 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.663+0000 7f5869caa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f58641992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:48.537 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.663+0000 7f5869caa700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f58641992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49210/0 (socket says 172.21.15.53:49210) 2023-07-14T19:17:48.537 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.663+0000 7f5869caa700 1 -- 172.21.15.53:0/2775934154 learned_addr learned my addr 172.21.15.53:0/2775934154 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:48.537 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.663+0000 7f5869caa700 1 -- 172.21.15.53:0/2775934154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5854009550 con 0x7f5864106060 2023-07-14T19:17:48.537 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.664+0000 7f5869caa700 1 --2- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f58641992c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5854000c00 tx=0x7f5854011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.664+0000 7f585affd700 1 -- 172.21.15.53:0/2775934154 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58540039f0 con 0x7f5864106060 2023-07-14T19:17:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.664+0000 7f585affd700 1 -- 172.21.15.53:0/2775934154 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5854003b50 con 0x7f5864106060 2023-07-14T19:17:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.664+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5864199a60 con 0x7f5864106060 2023-07-14T19:17:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.664+0000 7f585affd700 1 -- 172.21.15.53:0/2775934154 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f585401a780 con 0x7f5864106060 2023-07-14T19:17:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.664+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5864199ee0 con 0x7f5864106060 2023-07-14T19:17:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.665+0000 7f585affd700 1 -- 172.21.15.53:0/2775934154 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f5854024440 con 0x7f5864106060 2023-07-14T19:17:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.665+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58641932a0 con 0x7f5864106060 2023-07-14T19:17:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.665+0000 7f585affd700 1 --2- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5850038760 0x7f585003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:48.539 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.665+0000 7f585affd700 1 -- 172.21.15.53:0/2775934154 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f585404c8d0 con 0x7f5864106060 2023-07-14T19:17:48.539 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.666+0000 7f58694a9700 1 --2- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5850038760 0x7f585003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:48.539 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.666+0000 7f58694a9700 1 --2- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5850038760 0x7f585003ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5860006fd0 tx=0x7f5860006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:48.539 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:47.670+0000 7f585affd700 1 -- 172.21.15.53:0/2775934154 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f58540189a0 con 0x7f5864106060 2023-07-14T19:17:48.539 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.025+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f586402c8e0 con 0x7f5864106060 2023-07-14T19:17:48.539 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.026+0000 7f585affd700 1 -- 172.21.15.53:0/2775934154 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5854018350 con 0x7f5864106060 2023-07-14T19:17:48.539 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5850038760 msgr2=0x7f585003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:48.540 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 --2- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5850038760 0x7f585003ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5860006fd0 tx=0x7f5860006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:48.540 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 msgr2=0x7f58641992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:48.540 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 --2- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f58641992c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5854000c00 tx=0x7f5854011a30 comp rx=0 tx=0).stop 2023-07-14T19:17:48.540 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 shutdown_connections 2023-07-14T19:17:48.540 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 --2- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5850038760 0x7f585003ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:48.540 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 --2- 172.21.15.53:0/2775934154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5864106060 0x7f58641992c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:48.540 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 >> 172.21.15.53:0/2775934154 conn(0x7f58640ffac0 msgr2=0x7f5864101230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:48.541 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 shutdown_connections 2023-07-14T19:17:48.541 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:48.028+0000 7f586bf0e700 1 -- 172.21.15.53:0/2775934154 wait complete. 2023-07-14T19:17:49.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:49 smithi053 ceph-mon[96565]: mgrmap e13: x(active, since 6s) 2023-07-14T19:17:49.979 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.000+0000 7fa85fa74700 1 Processor -- start 2023-07-14T19:17:49.979 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.000+0000 7fa85fa74700 1 -- start start 2023-07-14T19:17:49.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85fa74700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa858105920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:49.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85fa74700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa858105e60 con 0x7fa858105510 2023-07-14T19:17:49.980 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85d810700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa858105920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:49.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85d810700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa858105920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49214/0 (socket says 172.21.15.53:49214) 2023-07-14T19:17:49.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85d810700 1 -- 172.21.15.53:0/1745491151 learned_addr learned my addr 172.21.15.53:0/1745491151 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:49.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85d810700 1 -- 172.21.15.53:0/1745491151 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa858105fa0 con 0x7fa858105510 2023-07-14T19:17:49.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85d810700 1 --2- 172.21.15.53:0/1745491151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa858105920 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa8480098a0 tx=0x7fa848009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c442122c8dac9a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:49.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85c80e700 1 -- 172.21.15.53:0/1745491151 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa848004160 con 0x7fa858105510 2023-07-14T19:17:49.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85c80e700 1 -- 172.21.15.53:0/1745491151 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa8480042c0 con 0x7fa858105510 2023-07-14T19:17:49.981 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.001+0000 7fa85c80e700 1 -- 172.21.15.53:0/1745491151 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8480119b0 con 0x7fa858105510 2023-07-14T19:17:49.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.002+0000 7fa85fa74700 1 -- 172.21.15.53:0/1745491151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 msgr2=0x7fa858105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:49.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.002+0000 7fa85fa74700 1 --2- 172.21.15.53:0/1745491151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa858105920 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa8480098a0 tx=0x7fa848009bb0 comp rx=0 tx=0).stop 2023-07-14T19:17:49.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.002+0000 7fa85fa74700 1 -- 172.21.15.53:0/1745491151 shutdown_connections 2023-07-14T19:17:49.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.002+0000 7fa85fa74700 1 --2- 172.21.15.53:0/1745491151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa858105920 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:49.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.002+0000 7fa85fa74700 1 -- 172.21.15.53:0/1745491151 >> 172.21.15.53:0/1745491151 conn(0x7fa858100bc0 msgr2=0x7fa858102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:49.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.002+0000 7fa85fa74700 1 -- 172.21.15.53:0/1745491151 shutdown_connections 2023-07-14T19:17:49.982 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.002+0000 7fa85fa74700 1 -- 172.21.15.53:0/1745491151 wait complete. 2023-07-14T19:17:49.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.003+0000 7fa85fa74700 1 Processor -- start 2023-07-14T19:17:49.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.003+0000 7fa85fa74700 1 -- start start 2023-07-14T19:17:49.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.003+0000 7fa85fa74700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa8581992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:49.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.003+0000 7fa85fa74700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa858199830 con 0x7fa858105510 2023-07-14T19:17:49.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.003+0000 7fa85d810700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa8581992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:49.983 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.003+0000 7fa85d810700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa8581992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49216/0 (socket says 172.21.15.53:49216) 2023-07-14T19:17:49.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.003+0000 7fa85d810700 1 -- 172.21.15.53:0/2682323234 learned_addr learned my addr 172.21.15.53:0/2682323234 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:49.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.004+0000 7fa85d810700 1 -- 172.21.15.53:0/2682323234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa848009550 con 0x7fa858105510 2023-07-14T19:17:49.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.004+0000 7fa85d810700 1 --2- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa8581992f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa848011980 tx=0x7fa848011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:49.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.004+0000 7fa84effd700 1 -- 172.21.15.53:0/2682323234 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa848003a60 con 0x7fa858105510 2023-07-14T19:17:49.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.004+0000 7fa84effd700 1 -- 172.21.15.53:0/2682323234 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa848003bc0 con 0x7fa858105510 2023-07-14T19:17:49.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.004+0000 7fa84effd700 1 -- 172.21.15.53:0/2682323234 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa84801a670 con 0x7fa858105510 2023-07-14T19:17:49.984 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.004+0000 7fa85fa74700 1 -- 172.21.15.53:0/2682323234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa858199a30 con 0x7fa858105510 2023-07-14T19:17:49.985 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.004+0000 7fa85fa74700 1 -- 172.21.15.53:0/2682323234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa858199e50 con 0x7fa858105510 2023-07-14T19:17:49.985 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.005+0000 7fa84effd700 1 -- 172.21.15.53:0/2682323234 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa84801a930 con 0x7fa858105510 2023-07-14T19:17:49.985 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.005+0000 7fa84cff9700 1 -- 172.21.15.53:0/2682323234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8400052f0 con 0x7fa858105510 2023-07-14T19:17:49.985 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.005+0000 7fa84effd700 1 --2- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa844038800 0x7fa84403acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:49.985 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.005+0000 7fa84effd700 1 -- 172.21.15.53:0/2682323234 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa84804cc30 con 0x7fa858105510 2023-07-14T19:17:49.985 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.006+0000 7fa85d00f700 1 --2- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa844038800 0x7fa84403acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:49.985 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.006+0000 7fa85d00f700 1 --2- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa844038800 0x7fa84403acb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa854006fd0 tx=0x7fa854006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:49.986 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.010+0000 7fa84effd700 1 -- 172.21.15.53:0/2682323234 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa8480189a0 con 0x7fa858105510 2023-07-14T19:17:49.986 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.580+0000 7fa84cff9700 1 -- 172.21.15.53:0/2682323234 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fa840005f40 con 0x7fa858105510 2023-07-14T19:17:49.986 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.582+0000 7fa84effd700 1 -- 172.21.15.53:0/2682323234 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fa8480187c0 con 0x7fa858105510 2023-07-14T19:17:49.986 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 -- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa844038800 msgr2=0x7fa84403acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:49.986 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 --2- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa844038800 0x7fa84403acb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa854006fd0 tx=0x7fa854006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:49.986 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 -- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 msgr2=0x7fa8581992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:49.986 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 --2- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa8581992f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa848011980 tx=0x7fa848011a60 comp rx=0 tx=0).stop 2023-07-14T19:17:49.986 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 -- 172.21.15.53:0/2682323234 shutdown_connections 2023-07-14T19:17:49.987 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 --2- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa844038800 0x7fa84403acb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:49.987 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 --2- 172.21.15.53:0/2682323234 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa858105510 0x7fa8581992f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:49.987 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 -- 172.21.15.53:0/2682323234 >> 172.21.15.53:0/2682323234 conn(0x7fa858100bc0 msgr2=0x7fa8581022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:49.987 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 -- 172.21.15.53:0/2682323234 shutdown_connections 2023-07-14T19:17:49.987 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr 2023-07-14T19:17:49.584+0000 7fa84cff9700 1 -- 172.21.15.53:0/2682323234 wait complete. 2023-07-14T19:17:49.987 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-14T19:17:49.987 INFO:teuthology.orchestra.run.smithi053.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-14T19:17:49.988 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:49.988 INFO:teuthology.orchestra.run.smithi053.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:17:49.988 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:49.988 INFO:teuthology.orchestra.run.smithi053.stdout:Or, if you are only running a single cluster on this host: 2023-07-14T19:17:49.988 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:49.988 INFO:teuthology.orchestra.run.smithi053.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-14T19:17:49.988 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:49.989 INFO:teuthology.orchestra.run.smithi053.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-14T19:17:49.989 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:49.989 INFO:teuthology.orchestra.run.smithi053.stdout: ceph telemetry on 2023-07-14T19:17:49.989 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:49.989 INFO:teuthology.orchestra.run.smithi053.stdout:For more information see: 2023-07-14T19:17:49.989 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:49.990 INFO:teuthology.orchestra.run.smithi053.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-14T19:17:49.990 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:17:49.990 INFO:teuthology.orchestra.run.smithi053.stdout:Bootstrap complete. 2023-07-14T19:17:50.012 INFO:tasks.cephadm:Fetching config... 2023-07-14T19:17:50.013 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:17:50.013 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-14T19:17:50.030 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-14T19:17:50.030 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:17:50.030 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-14T19:17:50.088 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-14T19:17:50.089 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:17:50.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-14T19:17:50.157 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-14T19:17:50.157 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:17:50.158 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-14T19:17:50.214 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-14T19:17:50.215 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChapZxBNZKdrzqvQpUT/4xiLDyvur3Bnvzfj8qJCG6gZVdrHaT6wbHXlGQxAX4f3bUzERuKe+yAS+uMKC4TOJP8JPh8kbbTJkQHczIEydEGXUsZe/H4kjrFsTD/pfS0KcRlwSAs1eLWZcJ3nAW+l2BPuaiGf1hTItTZ5T2DhDmYGFMEobg4xze24Fpoxes0PiZZFJ/jKsMku4G0kIqK6o8V4csVhf0SBDM5t+LHTD5jMBLlt/p0LCfQ9f39N6y0cwNVoF+ZGivCkASannUt6np/EFQBIS9Q08hd9zXQnstYsdeMUAOoq/XsQpaxDR+K9LCLfs8zPEAvKfgRGyfIPO3tOk75oPthF0wzYsHKfd4AQpw6xXiK1qLyAcnYTtl8Ri6In+k/EopYbWnd8S57ZhOWInYP7vBbrdsUP2yMPQKCCagkZSTQJ0HeGRMFsoBJjR8LKkCcRnCssiMO+AAMABcJjOxpqmJxoRn5dgS7oI+q1s8Z2CJ627jXhxpvkifeLs= ceph-c5147628-227a-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T19:17:50.296 INFO:teuthology.orchestra.run.smithi053.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChapZxBNZKdrzqvQpUT/4xiLDyvur3Bnvzfj8qJCG6gZVdrHaT6wbHXlGQxAX4f3bUzERuKe+yAS+uMKC4TOJP8JPh8kbbTJkQHczIEydEGXUsZe/H4kjrFsTD/pfS0KcRlwSAs1eLWZcJ3nAW+l2BPuaiGf1hTItTZ5T2DhDmYGFMEobg4xze24Fpoxes0PiZZFJ/jKsMku4G0kIqK6o8V4csVhf0SBDM5t+LHTD5jMBLlt/p0LCfQ9f39N6y0cwNVoF+ZGivCkASannUt6np/EFQBIS9Q08hd9zXQnstYsdeMUAOoq/XsQpaxDR+K9LCLfs8zPEAvKfgRGyfIPO3tOk75oPthF0wzYsHKfd4AQpw6xXiK1qLyAcnYTtl8Ri6In+k/EopYbWnd8S57ZhOWInYP7vBbrdsUP2yMPQKCCagkZSTQJ0HeGRMFsoBJjR8LKkCcRnCssiMO+AAMABcJjOxpqmJxoRn5dgS7oI+q1s8Z2CJ627jXhxpvkifeLs= ceph-c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:17:50.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChapZxBNZKdrzqvQpUT/4xiLDyvur3Bnvzfj8qJCG6gZVdrHaT6wbHXlGQxAX4f3bUzERuKe+yAS+uMKC4TOJP8JPh8kbbTJkQHczIEydEGXUsZe/H4kjrFsTD/pfS0KcRlwSAs1eLWZcJ3nAW+l2BPuaiGf1hTItTZ5T2DhDmYGFMEobg4xze24Fpoxes0PiZZFJ/jKsMku4G0kIqK6o8V4csVhf0SBDM5t+LHTD5jMBLlt/p0LCfQ9f39N6y0cwNVoF+ZGivCkASannUt6np/EFQBIS9Q08hd9zXQnstYsdeMUAOoq/XsQpaxDR+K9LCLfs8zPEAvKfgRGyfIPO3tOk75oPthF0wzYsHKfd4AQpw6xXiK1qLyAcnYTtl8Ri6In+k/EopYbWnd8S57ZhOWInYP7vBbrdsUP2yMPQKCCagkZSTQJ0HeGRMFsoBJjR8LKkCcRnCssiMO+AAMABcJjOxpqmJxoRn5dgS7oI+q1s8Z2CJ627jXhxpvkifeLs= ceph-c5147628-227a-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T19:17:50.356 INFO:teuthology.orchestra.run.smithi070.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChapZxBNZKdrzqvQpUT/4xiLDyvur3Bnvzfj8qJCG6gZVdrHaT6wbHXlGQxAX4f3bUzERuKe+yAS+uMKC4TOJP8JPh8kbbTJkQHczIEydEGXUsZe/H4kjrFsTD/pfS0KcRlwSAs1eLWZcJ3nAW+l2BPuaiGf1hTItTZ5T2DhDmYGFMEobg4xze24Fpoxes0PiZZFJ/jKsMku4G0kIqK6o8V4csVhf0SBDM5t+LHTD5jMBLlt/p0LCfQ9f39N6y0cwNVoF+ZGivCkASannUt6np/EFQBIS9Q08hd9zXQnstYsdeMUAOoq/XsQpaxDR+K9LCLfs8zPEAvKfgRGyfIPO3tOk75oPthF0wzYsHKfd4AQpw6xXiK1qLyAcnYTtl8Ri6In+k/EopYbWnd8S57ZhOWInYP7vBbrdsUP2yMPQKCCagkZSTQJ0HeGRMFsoBJjR8LKkCcRnCssiMO+AAMABcJjOxpqmJxoRn5dgS7oI+q1s8Z2CJ627jXhxpvkifeLs= ceph-c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:17:50.368 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChapZxBNZKdrzqvQpUT/4xiLDyvur3Bnvzfj8qJCG6gZVdrHaT6wbHXlGQxAX4f3bUzERuKe+yAS+uMKC4TOJP8JPh8kbbTJkQHczIEydEGXUsZe/H4kjrFsTD/pfS0KcRlwSAs1eLWZcJ3nAW+l2BPuaiGf1hTItTZ5T2DhDmYGFMEobg4xze24Fpoxes0PiZZFJ/jKsMku4G0kIqK6o8V4csVhf0SBDM5t+LHTD5jMBLlt/p0LCfQ9f39N6y0cwNVoF+ZGivCkASannUt6np/EFQBIS9Q08hd9zXQnstYsdeMUAOoq/XsQpaxDR+K9LCLfs8zPEAvKfgRGyfIPO3tOk75oPthF0wzYsHKfd4AQpw6xXiK1qLyAcnYTtl8Ri6In+k/EopYbWnd8S57ZhOWInYP7vBbrdsUP2yMPQKCCagkZSTQJ0HeGRMFsoBJjR8LKkCcRnCssiMO+AAMABcJjOxpqmJxoRn5dgS7oI+q1s8Z2CJ627jXhxpvkifeLs= ceph-c5147628-227a-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T19:17:50.416 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChapZxBNZKdrzqvQpUT/4xiLDyvur3Bnvzfj8qJCG6gZVdrHaT6wbHXlGQxAX4f3bUzERuKe+yAS+uMKC4TOJP8JPh8kbbTJkQHczIEydEGXUsZe/H4kjrFsTD/pfS0KcRlwSAs1eLWZcJ3nAW+l2BPuaiGf1hTItTZ5T2DhDmYGFMEobg4xze24Fpoxes0PiZZFJ/jKsMku4G0kIqK6o8V4csVhf0SBDM5t+LHTD5jMBLlt/p0LCfQ9f39N6y0cwNVoF+ZGivCkASannUt6np/EFQBIS9Q08hd9zXQnstYsdeMUAOoq/XsQpaxDR+K9LCLfs8zPEAvKfgRGyfIPO3tOk75oPthF0wzYsHKfd4AQpw6xXiK1qLyAcnYTtl8Ri6In+k/EopYbWnd8S57ZhOWInYP7vBbrdsUP2yMPQKCCagkZSTQJ0HeGRMFsoBJjR8LKkCcRnCssiMO+AAMABcJjOxpqmJxoRn5dgS7oI+q1s8Z2CJ627jXhxpvkifeLs= ceph-c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:17:50.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-14T19:17:50.847 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:50 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2682323234' entity='client.admin' 2023-07-14T19:17:51.201 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:52.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:52.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:52.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:17:52.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:52.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:52.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:52.959 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.955+0000 7ff77da8c700 1 -- 172.21.15.53:0/3505591065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 msgr2=0x7ff7780ffb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:52.959 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.955+0000 7ff77da8c700 1 --2- 172.21.15.53:0/3505591065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 0x7ff7780ffb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff768009910 tx=0x7ff768009c20 comp rx=0 tx=0).stop 2023-07-14T19:17:52.960 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.955+0000 7ff77da8c700 1 -- 172.21.15.53:0/3505591065 shutdown_connections 2023-07-14T19:17:52.960 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.955+0000 7ff77da8c700 1 --2- 172.21.15.53:0/3505591065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 0x7ff7780ffb30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:52.960 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.955+0000 7ff77da8c700 1 -- 172.21.15.53:0/3505591065 >> 172.21.15.53:0/3505591065 conn(0x7ff7780facb0 msgr2=0x7ff7780fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:52.960 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.955+0000 7ff77da8c700 1 -- 172.21.15.53:0/3505591065 shutdown_connections 2023-07-14T19:17:52.960 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.955+0000 7ff77da8c700 1 -- 172.21.15.53:0/3505591065 wait complete. 2023-07-14T19:17:52.960 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.956+0000 7ff77da8c700 1 Processor -- start 2023-07-14T19:17:52.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.956+0000 7ff77da8c700 1 -- start start 2023-07-14T19:17:52.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.956+0000 7ff77da8c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 0x7ff778194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:52.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.956+0000 7ff77da8c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff778194dd0 con 0x7ff7780ff720 2023-07-14T19:17:52.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.956+0000 7ff7777fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 0x7ff778194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:52.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.956+0000 7ff7777fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 0x7ff778194890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49222/0 (socket says 172.21.15.53:49222) 2023-07-14T19:17:52.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.956+0000 7ff7777fe700 1 -- 172.21.15.53:0/3732525912 learned_addr learned my addr 172.21.15.53:0/3732525912 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:52.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.956+0000 7ff7777fe700 1 -- 172.21.15.53:0/3732525912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7680095f0 con 0x7ff7780ff720 2023-07-14T19:17:52.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.956+0000 7ff7777fe700 1 --2- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 0x7ff778194890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff768006010 tx=0x7ff7680057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:52.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.957+0000 7ff774ff9700 1 -- 172.21.15.53:0/3732525912 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff76801d070 con 0x7ff7780ff720 2023-07-14T19:17:52.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.957+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff778194fd0 con 0x7ff7780ff720 2023-07-14T19:17:52.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.957+0000 7ff774ff9700 1 -- 172.21.15.53:0/3732525912 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7680044e0 con 0x7ff7780ff720 2023-07-14T19:17:52.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.957+0000 7ff774ff9700 1 -- 172.21.15.53:0/3732525912 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7680039b0 con 0x7ff7780ff720 2023-07-14T19:17:52.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.957+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff778195410 con 0x7ff7780ff720 2023-07-14T19:17:52.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.957+0000 7ff774ff9700 1 -- 172.21.15.53:0/3732525912 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7ff768003c70 con 0x7ff7780ff720 2023-07-14T19:17:52.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.958+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77818e2f0 con 0x7ff7780ff720 2023-07-14T19:17:52.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.958+0000 7ff774ff9700 1 --2- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff7600387b0 0x7ff76003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:52.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.958+0000 7ff774ff9700 1 -- 172.21.15.53:0/3732525912 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff76804c820 con 0x7ff7780ff720 2023-07-14T19:17:52.964 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.958+0000 7ff776ffd700 1 --2- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff7600387b0 0x7ff76003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:52.964 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.958+0000 7ff776ffd700 1 --2- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff7600387b0 0x7ff76003ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff76c006fd0 tx=0x7ff76c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:52.965 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:52.962+0000 7ff774ff9700 1 -- 172.21.15.53:0/3732525912 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff76802a950 con 0x7ff7780ff720 2023-07-14T19:17:53.163 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.160+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7ff77802c8e0 con 0x7ff7780ff720 2023-07-14T19:17:53.166 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.163+0000 7ff774ff9700 1 -- 172.21.15.53:0/3732525912 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff76802a360 con 0x7ff7780ff720 2023-07-14T19:17:53.168 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.165+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff7600387b0 msgr2=0x7ff76003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:53.168 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.165+0000 7ff77da8c700 1 --2- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff7600387b0 0x7ff76003ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff76c006fd0 tx=0x7ff76c006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.165+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 msgr2=0x7ff778194890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.165+0000 7ff77da8c700 1 --2- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 0x7ff778194890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff768006010 tx=0x7ff7680057c0 comp rx=0 tx=0).stop 2023-07-14T19:17:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.166+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 shutdown_connections 2023-07-14T19:17:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.166+0000 7ff77da8c700 1 --2- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff7600387b0 0x7ff76003ac60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.166+0000 7ff77da8c700 1 --2- 172.21.15.53:0/3732525912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7780ff720 0x7ff778194890 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.166+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 >> 172.21.15.53:0/3732525912 conn(0x7ff7780facb0 msgr2=0x7ff7780fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:53.170 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.166+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 shutdown_connections 2023-07-14T19:17:53.170 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:53.166+0000 7ff77da8c700 1 -- 172.21.15.53:0/3732525912 wait complete. 2023-07-14T19:17:53.558 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-14T19:17:53.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-14T19:17:54.116 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:54.532 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:54 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3732525912' entity='client.admin' 2023-07-14T19:17:54.532 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:54.532 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:55.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.301+0000 7fd3a80d4700 1 -- 172.21.15.53:0/1956203660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 msgr2=0x7fd3a00fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:55.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.301+0000 7fd3a80d4700 1 --2- 172.21.15.53:0/1956203660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 0x7fd3a00fea20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009910 tx=0x7fd39c009c20 comp rx=0 tx=0).stop 2023-07-14T19:17:55.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.301+0000 7fd3a80d4700 1 -- 172.21.15.53:0/1956203660 shutdown_connections 2023-07-14T19:17:55.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.301+0000 7fd3a80d4700 1 --2- 172.21.15.53:0/1956203660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 0x7fd3a00fea20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:55.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.301+0000 7fd3a80d4700 1 -- 172.21.15.53:0/1956203660 >> 172.21.15.53:0/1956203660 conn(0x7fd3a00f9ba0 msgr2=0x7fd3a00fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:55.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.301+0000 7fd3a80d4700 1 -- 172.21.15.53:0/1956203660 shutdown_connections 2023-07-14T19:17:55.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.301+0000 7fd3a80d4700 1 -- 172.21.15.53:0/1956203660 wait complete. 2023-07-14T19:17:55.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.302+0000 7fd3a80d4700 1 Processor -- start 2023-07-14T19:17:55.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.302+0000 7fd3a80d4700 1 -- start start 2023-07-14T19:17:55.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.302+0000 7fd3a80d4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 0x7fd3a0193780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:55.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.302+0000 7fd3a80d4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a0193cc0 con 0x7fd3a00fe610 2023-07-14T19:17:55.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.302+0000 7fd3a5e70700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 0x7fd3a0193780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:55.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.302+0000 7fd3a5e70700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 0x7fd3a0193780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49228/0 (socket says 172.21.15.53:49228) 2023-07-14T19:17:55.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.302+0000 7fd3a5e70700 1 -- 172.21.15.53:0/557854631 learned_addr learned my addr 172.21.15.53:0/557854631 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:55.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.302+0000 7fd3a5e70700 1 -- 172.21.15.53:0/557854631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c0095f0 con 0x7fd3a00fe610 2023-07-14T19:17:55.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.302+0000 7fd3a5e70700 1 --2- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 0x7fd3a0193780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd39c006010 tx=0x7fd39c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:55.308 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.303+0000 7fd392ffd700 1 -- 172.21.15.53:0/557854631 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd39c01d070 con 0x7fd3a00fe610 2023-07-14T19:17:55.308 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.303+0000 7fd392ffd700 1 -- 172.21.15.53:0/557854631 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd39c0044e0 con 0x7fd3a00fe610 2023-07-14T19:17:55.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.303+0000 7fd392ffd700 1 -- 172.21.15.53:0/557854631 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd39c0039b0 con 0x7fd3a00fe610 2023-07-14T19:17:55.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.303+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a0193ec0 con 0x7fd3a00fe610 2023-07-14T19:17:55.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.303+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a0194300 con 0x7fd3a00fe610 2023-07-14T19:17:55.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.303+0000 7fd392ffd700 1 -- 172.21.15.53:0/557854631 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fd39c003c20 con 0x7fd3a00fe610 2023-07-14T19:17:55.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.304+0000 7fd392ffd700 1 --2- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd38c038800 0x7fd38c03acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:55.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.304+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a018d1e0 con 0x7fd3a00fe610 2023-07-14T19:17:55.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.304+0000 7fd392ffd700 1 -- 172.21.15.53:0/557854631 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd39c04ca00 con 0x7fd3a00fe610 2023-07-14T19:17:55.310 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.304+0000 7fd3a566f700 1 --2- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd38c038800 0x7fd38c03acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:55.310 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.304+0000 7fd3a566f700 1 --2- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd38c038800 0x7fd38c03acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd394006fd0 tx=0x7fd394006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:55.311 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.309+0000 7fd392ffd700 1 -- 172.21.15.53:0/557854631 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd39c00f460 con 0x7fd3a00fe610 2023-07-14T19:17:55.510 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.508+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3a0060b10 con 0x7fd38c038800 2023-07-14T19:17:55.514 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.511+0000 7fd392ffd700 1 -- 172.21.15.53:0/557854631 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd3a0060b10 con 0x7fd38c038800 2023-07-14T19:17:55.516 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd38c038800 msgr2=0x7fd38c03acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:55.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 --2- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd38c038800 0x7fd38c03acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd394006fd0 tx=0x7fd394006e40 comp rx=0 tx=0).stop 2023-07-14T19:17:55.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 msgr2=0x7fd3a0193780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:55.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 --2- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 0x7fd3a0193780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd39c006010 tx=0x7fd39c0057c0 comp rx=0 tx=0).stop 2023-07-14T19:17:55.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 shutdown_connections 2023-07-14T19:17:55.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 --2- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd38c038800 0x7fd38c03acb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:55.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 --2- 172.21.15.53:0/557854631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3a00fe610 0x7fd3a0193780 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:55.518 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 >> 172.21.15.53:0/557854631 conn(0x7fd3a00f9ba0 msgr2=0x7fd3a00fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:55.518 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 shutdown_connections 2023-07-14T19:17:55.518 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:55.514+0000 7fd3a80d4700 1 -- 172.21.15.53:0/557854631 wait complete. 2023-07-14T19:17:55.965 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi070 2023-07-14T19:17:55.965 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:17:55.966 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/etc/ceph/ceph.conf 2023-07-14T19:17:55.986 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:17:55.986 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:17:56.045 INFO:tasks.cephadm:Adding host smithi070 to orchestrator... 2023-07-14T19:17:56.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch host add smithi070 2023-07-14T19:17:56.613 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:17:56.813 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:56 smithi053 ceph-mon[96565]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:17:56.813 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:56 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:56.813 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:56 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:17:56.813 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:56 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:17:56.813 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:56 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:17:56.814 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:56 smithi053 ceph-mon[96565]: Updating smithi053:/etc/ceph/ceph.conf 2023-07-14T19:17:56.814 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:56 smithi053 ceph-mon[96565]: Updating smithi053:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:17:56.814 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:56 smithi053 ceph-mon[96565]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:17:57.716 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:57 smithi053 ceph-mon[96565]: Updating smithi053:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:17:57.716 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:57.716 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:57.717 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:17:57.925 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.922+0000 7f28f3774700 1 -- 172.21.15.53:0/899626802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 msgr2=0x7f28ec104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:17:57.925 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.922+0000 7f28f3774700 1 --2- 172.21.15.53:0/899626802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 0x7f28ec104a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009910 tx=0x7f28e8009c20 comp rx=0 tx=0).stop 2023-07-14T19:17:57.925 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.922+0000 7f28f3774700 1 -- 172.21.15.53:0/899626802 shutdown_connections 2023-07-14T19:17:57.925 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.922+0000 7f28f3774700 1 --2- 172.21.15.53:0/899626802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 0x7f28ec104a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:17:57.925 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.922+0000 7f28f3774700 1 -- 172.21.15.53:0/899626802 >> 172.21.15.53:0/899626802 conn(0x7f28ec0743e0 msgr2=0x7f28ec0747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:17:57.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.923+0000 7f28f3774700 1 -- 172.21.15.53:0/899626802 shutdown_connections 2023-07-14T19:17:57.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.923+0000 7f28f3774700 1 -- 172.21.15.53:0/899626802 wait complete. 2023-07-14T19:17:57.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.923+0000 7f28f3774700 1 Processor -- start 2023-07-14T19:17:57.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.923+0000 7f28f3774700 1 -- start start 2023-07-14T19:17:57.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28f3774700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 0x7f28ec19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:57.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28f3774700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec19ae50 con 0x7f28ec104660 2023-07-14T19:17:57.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28f1510700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 0x7f28ec19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:57.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28f1510700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 0x7f28ec19a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49234/0 (socket says 172.21.15.53:49234) 2023-07-14T19:17:57.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28f1510700 1 -- 172.21.15.53:0/4344047 learned_addr learned my addr 172.21.15.53:0/4344047 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:17:57.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28f1510700 1 -- 172.21.15.53:0/4344047 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e80095f0 con 0x7f28ec104660 2023-07-14T19:17:57.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28f1510700 1 --2- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 0x7f28ec19a910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f28e8004f40 tx=0x7f28e8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:57.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28de7fc700 1 -- 172.21.15.53:0/4344047 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28e801d070 con 0x7f28ec104660 2023-07-14T19:17:57.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28de7fc700 1 -- 172.21.15.53:0/4344047 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28e8004370 con 0x7f28ec104660 2023-07-14T19:17:57.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ec19b0b0 con 0x7f28ec104660 2023-07-14T19:17:57.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28de7fc700 1 -- 172.21.15.53:0/4344047 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28e80039b0 con 0x7f28ec104660 2023-07-14T19:17:57.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.924+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ec194990 con 0x7f28ec104660 2023-07-14T19:17:57.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.925+0000 7f28de7fc700 1 -- 172.21.15.53:0/4344047 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f28e8003c20 con 0x7f28ec104660 2023-07-14T19:17:57.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.925+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ec061c70 con 0x7f28ec104660 2023-07-14T19:17:57.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.925+0000 7f28de7fc700 1 --2- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f28d8038800 0x7f28d803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:17:57.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.925+0000 7f28de7fc700 1 -- 172.21.15.53:0/4344047 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f28e804c750 con 0x7f28ec104660 2023-07-14T19:17:57.930 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.926+0000 7f28f0d0f700 1 --2- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f28d8038800 0x7f28d803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:17:57.930 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.926+0000 7f28f0d0f700 1 --2- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f28d8038800 0x7f28d803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f28e0006fd0 tx=0x7f28e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:17:57.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:57.930+0000 7f28de7fc700 1 -- 172.21.15.53:0/4344047 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f28e8029360 con 0x7f28ec104660 2023-07-14T19:17:58.131 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:17:58.129+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi070", "target": ["mon-mgr", ""]}) v1 -- 0x7f28ec067f50 con 0x7f28d8038800 2023-07-14T19:17:59.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:58 smithi053 ceph-mon[96565]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi070", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:00.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:17:59 smithi053 ceph-mon[96565]: Deploying cephadm binary to smithi070 2023-07-14T19:18:00.895 INFO:teuthology.orchestra.run.smithi053.stdout:Added host 'smithi070' with addr '172.21.15.70' 2023-07-14T19:18:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.893+0000 7f28de7fc700 1 -- 172.21.15.53:0/4344047 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f28ec067f50 con 0x7f28d8038800 2023-07-14T19:18:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.895+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f28d8038800 msgr2=0x7f28d803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.895+0000 7f28f3774700 1 --2- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f28d8038800 0x7f28d803acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f28e0006fd0 tx=0x7f28e0006e40 comp rx=0 tx=0).stop 2023-07-14T19:18:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.895+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 msgr2=0x7f28ec19a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.895+0000 7f28f3774700 1 --2- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 0x7f28ec19a910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f28e8004f40 tx=0x7f28e8005020 comp rx=0 tx=0).stop 2023-07-14T19:18:00.898 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.895+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 shutdown_connections 2023-07-14T19:18:00.898 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.895+0000 7f28f3774700 1 --2- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f28d8038800 0x7f28d803acb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:00.898 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.895+0000 7f28f3774700 1 --2- 172.21.15.53:0/4344047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec104660 0x7f28ec19a910 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:00.898 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.895+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 >> 172.21.15.53:0/4344047 conn(0x7f28ec0743e0 msgr2=0x7f28ec067840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:00.898 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.896+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 shutdown_connections 2023-07-14T19:18:00.898 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:00.896+0000 7f28f3774700 1 -- 172.21.15.53:0/4344047 wait complete. 2023-07-14T19:18:01.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T19:18:01.836 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:18:02.225 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:01 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:02.226 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:01 smithi053 ceph-mon[96565]: Added host smithi070 2023-07-14T19:18:02.226 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:01 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:02.226 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:01 smithi053 ceph-mon[96565]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:02.226 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:01 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:03.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.047+0000 7f20c0189700 1 -- 172.21.15.53:0/2171141404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 msgr2=0x7f20b8105b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:03.052 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.047+0000 7f20c0189700 1 --2- 172.21.15.53:0/2171141404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 0x7f20b8105b90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009910 tx=0x7f20b4009c20 comp rx=0 tx=0).stop 2023-07-14T19:18:03.052 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.047+0000 7f20c0189700 1 -- 172.21.15.53:0/2171141404 shutdown_connections 2023-07-14T19:18:03.052 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.047+0000 7f20c0189700 1 --2- 172.21.15.53:0/2171141404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 0x7f20b8105b90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:03.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.047+0000 7f20c0189700 1 -- 172.21.15.53:0/2171141404 >> 172.21.15.53:0/2171141404 conn(0x7f20b80743e0 msgr2=0x7f20b80747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:03.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.047+0000 7f20c0189700 1 -- 172.21.15.53:0/2171141404 shutdown_connections 2023-07-14T19:18:03.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.048+0000 7f20c0189700 1 -- 172.21.15.53:0/2171141404 wait complete. 2023-07-14T19:18:03.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.048+0000 7f20c0189700 1 Processor -- start 2023-07-14T19:18:03.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.048+0000 7f20c0189700 1 -- start start 2023-07-14T19:18:03.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.048+0000 7f20c0189700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 0x7f20b819a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:03.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.048+0000 7f20c0189700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b819aeb0 con 0x7f20b81057c0 2023-07-14T19:18:03.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.048+0000 7f20bdf25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 0x7f20b819a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:03.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.048+0000 7f20bdf25700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 0x7f20b819a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49260/0 (socket says 172.21.15.53:49260) 2023-07-14T19:18:03.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.048+0000 7f20bdf25700 1 -- 172.21.15.53:0/1693802518 learned_addr learned my addr 172.21.15.53:0/1693802518 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:18:03.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.049+0000 7f20bdf25700 1 -- 172.21.15.53:0/1693802518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b40095f0 con 0x7f20b81057c0 2023-07-14T19:18:03.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.049+0000 7f20bdf25700 1 --2- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 0x7f20b819a970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f20b4004f40 tx=0x7f20b4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:03.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.049+0000 7f20aeffd700 1 -- 172.21.15.53:0/1693802518 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20b401d070 con 0x7f20b81057c0 2023-07-14T19:18:03.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.049+0000 7f20aeffd700 1 -- 172.21.15.53:0/1693802518 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20b4004370 con 0x7f20b81057c0 2023-07-14T19:18:03.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.049+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b819b0b0 con 0x7f20b81057c0 2023-07-14T19:18:03.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.049+0000 7f20aeffd700 1 -- 172.21.15.53:0/1693802518 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20b40039b0 con 0x7f20b81057c0 2023-07-14T19:18:03.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.049+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b81949f0 con 0x7f20b81057c0 2023-07-14T19:18:03.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.050+0000 7f20aeffd700 1 -- 172.21.15.53:0/1693802518 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f20b4003c70 con 0x7f20b81057c0 2023-07-14T19:18:03.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.050+0000 7f20aeffd700 1 --2- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f20a40387b0 0x7f20a403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:03.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.050+0000 7f20aeffd700 1 -- 172.21.15.53:0/1693802518 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f20b404d450 con 0x7f20b81057c0 2023-07-14T19:18:03.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.050+0000 7f20bd724700 1 --2- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f20a40387b0 0x7f20a403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:03.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.051+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209c005320 con 0x7f20b81057c0 2023-07-14T19:18:03.060 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.051+0000 7f20bd724700 1 --2- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f20a40387b0 0x7f20a403ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f20a8006fd0 tx=0x7f20a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:03.060 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.056+0000 7f20aeffd700 1 -- 172.21.15.53:0/1693802518 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f20b4026020 con 0x7f20b81057c0 2023-07-14T19:18:03.258 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.256+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f209c000bf0 con 0x7f20a40387b0 2023-07-14T19:18:03.259 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.257+0000 7f20aeffd700 1 -- 172.21.15.53:0/1693802518 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f209c000bf0 con 0x7f20a40387b0 2023-07-14T19:18:03.259 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:18:03.259 INFO:teuthology.orchestra.run.smithi053.stdout:[{"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}] 2023-07-14T19:18:03.261 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f20a40387b0 msgr2=0x7f20a403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:03.261 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 --2- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f20a40387b0 0x7f20a403ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f20a8006fd0 tx=0x7f20a8006e40 comp rx=0 tx=0).stop 2023-07-14T19:18:03.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 msgr2=0x7f20b819a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:03.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 --2- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 0x7f20b819a970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f20b4004f40 tx=0x7f20b4005020 comp rx=0 tx=0).stop 2023-07-14T19:18:03.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 shutdown_connections 2023-07-14T19:18:03.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 --2- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f20a40387b0 0x7f20a403ac60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:03.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 --2- 172.21.15.53:0/1693802518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20b81057c0 0x7f20b819a970 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:03.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 >> 172.21.15.53:0/1693802518 conn(0x7f20b80743e0 msgr2=0x7f20b8106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:03.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 shutdown_connections 2023-07-14T19:18:03.263 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:03.260+0000 7f20c0189700 1 -- 172.21.15.53:0/1693802518 wait complete. 2023-07-14T19:18:03.281 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:02 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:03.694 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi104 2023-07-14T19:18:03.694 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:18:03.694 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.conf 2023-07-14T19:18:03.717 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:18:03.717 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:03.776 INFO:tasks.cephadm:Adding host smithi104 to orchestrator... 2023-07-14T19:18:03.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch host add smithi104 2023-07-14T19:18:04.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:03 smithi053 ceph-mon[96565]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:04.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:03 smithi053 ceph-mon[96565]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:18:04.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:03 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:04.335 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:18:05.567 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.564+0000 7fd6fd28f700 1 -- 172.21.15.53:0/1985174294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 msgr2=0x7fd6f8104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.564+0000 7fd6fd28f700 1 --2- 172.21.15.53:0/1985174294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 0x7fd6f8104a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009910 tx=0x7fd6ec009c20 comp rx=0 tx=0).stop 2023-07-14T19:18:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.564+0000 7fd6fd28f700 1 -- 172.21.15.53:0/1985174294 shutdown_connections 2023-07-14T19:18:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.564+0000 7fd6fd28f700 1 --2- 172.21.15.53:0/1985174294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 0x7fd6f8104a30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.564+0000 7fd6fd28f700 1 -- 172.21.15.53:0/1985174294 >> 172.21.15.53:0/1985174294 conn(0x7fd6f80743e0 msgr2=0x7fd6f80747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:05.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.564+0000 7fd6fd28f700 1 -- 172.21.15.53:0/1985174294 shutdown_connections 2023-07-14T19:18:05.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.564+0000 7fd6fd28f700 1 -- 172.21.15.53:0/1985174294 wait complete. 2023-07-14T19:18:05.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.565+0000 7fd6fd28f700 1 Processor -- start 2023-07-14T19:18:05.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.565+0000 7fd6fd28f700 1 -- start start 2023-07-14T19:18:05.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.565+0000 7fd6fd28f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 0x7fd6f8100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:05.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.565+0000 7fd6fd28f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f81021a0 con 0x7fd6f8104660 2023-07-14T19:18:05.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.565+0000 7fd6f6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 0x7fd6f8100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:05.570 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.565+0000 7fd6f6ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 0x7fd6f8100220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49266/0 (socket says 172.21.15.53:49266) 2023-07-14T19:18:05.570 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.565+0000 7fd6f6ffd700 1 -- 172.21.15.53:0/972151049 learned_addr learned my addr 172.21.15.53:0/972151049 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:18:05.570 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.566+0000 7fd6f6ffd700 1 -- 172.21.15.53:0/972151049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec0095f0 con 0x7fd6f8104660 2023-07-14T19:18:05.570 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.566+0000 7fd6f6ffd700 1 --2- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 0x7fd6f8100220 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec004f40 tx=0x7fd6ec005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:05.570 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.566+0000 7fd6dffff700 1 -- 172.21.15.53:0/972151049 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6ec01d070 con 0x7fd6f8104660 2023-07-14T19:18:05.570 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.566+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f8100760 con 0x7fd6f8104660 2023-07-14T19:18:05.570 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.566+0000 7fd6dffff700 1 -- 172.21.15.53:0/972151049 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6ec004370 con 0x7fd6f8104660 2023-07-14T19:18:05.571 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.566+0000 7fd6dffff700 1 -- 172.21.15.53:0/972151049 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6ec0039b0 con 0x7fd6f8104660 2023-07-14T19:18:05.571 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.566+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f8100be0 con 0x7fd6f8104660 2023-07-14T19:18:05.571 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.567+0000 7fd6dffff700 1 -- 172.21.15.53:0/972151049 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fd6ec003c70 con 0x7fd6f8104660 2023-07-14T19:18:05.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.567+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f8061c70 con 0x7fd6f8104660 2023-07-14T19:18:05.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.567+0000 7fd6dffff700 1 --2- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6e00387b0 0x7fd6e003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:05.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.567+0000 7fd6dffff700 1 -- 172.21.15.53:0/972151049 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd6ec04c780 con 0x7fd6f8104660 2023-07-14T19:18:05.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.567+0000 7fd6f67fc700 1 --2- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6e00387b0 0x7fd6e003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:05.573 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.568+0000 7fd6f67fc700 1 --2- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6e00387b0 0x7fd6e003ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8006fd0 tx=0x7fd6e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:05.573 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.572+0000 7fd6dffff700 1 -- 172.21.15.53:0/972151049 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd6ec02a7c0 con 0x7fd6f8104660 2023-07-14T19:18:05.773 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:05.772+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6f8101090 con 0x7fd6e00387b0 2023-07-14T19:18:05.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:05 smithi053 ceph-mon[96565]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:07.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:07 smithi053 ceph-mon[96565]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:07.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:07.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:07.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:07.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:07.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:18:07.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:07.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:07.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:07 smithi053 ceph-mon[96565]: Updating smithi070:/etc/ceph/ceph.conf 2023-07-14T19:18:08.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:08 smithi053 ceph-mon[96565]: Deploying cephadm binary to smithi104 2023-07-14T19:18:08.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:08 smithi053 ceph-mon[96565]: Updating smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:08.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:08 smithi053 ceph-mon[96565]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:08.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:08 smithi053 ceph-mon[96565]: Updating smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:18:08.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:08 smithi053 ceph-mon[96565]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:08.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:08.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:08.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:08.587 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.586+0000 7fd6dffff700 1 -- 172.21.15.53:0/972151049 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd6f8101090 con 0x7fd6e00387b0 2023-07-14T19:18:08.588 INFO:teuthology.orchestra.run.smithi053.stdout:Added host 'smithi104' with addr '172.21.15.104' 2023-07-14T19:18:08.590 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6e00387b0 msgr2=0x7fd6e003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:08.590 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 --2- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6e00387b0 0x7fd6e003ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8006fd0 tx=0x7fd6e8006e40 comp rx=0 tx=0).stop 2023-07-14T19:18:08.590 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 msgr2=0x7fd6f8100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:08.590 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 --2- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 0x7fd6f8100220 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec004f40 tx=0x7fd6ec005020 comp rx=0 tx=0).stop 2023-07-14T19:18:08.590 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 shutdown_connections 2023-07-14T19:18:08.590 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 --2- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6e00387b0 0x7fd6e003ac60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:08.590 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 --2- 172.21.15.53:0/972151049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6f8104660 0x7fd6f8100220 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:08.591 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 >> 172.21.15.53:0/972151049 conn(0x7fd6f80743e0 msgr2=0x7fd6f80680d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:08.591 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 shutdown_connections 2023-07-14T19:18:08.591 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:08.588+0000 7fd6fd28f700 1 -- 172.21.15.53:0/972151049 wait complete. 2023-07-14T19:18:09.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T19:18:09.563 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:18:09.962 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:09.962 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:09 smithi053 ceph-mon[96565]: Added host smithi104 2023-07-14T19:18:09.962 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:09.963 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:09.963 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:09 smithi053 ceph-mon[96565]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:10.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.780+0000 7f48f602a700 1 -- 172.21.15.53:0/1376961224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 msgr2=0x7f48f00ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:10.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.780+0000 7f48f602a700 1 --2- 172.21.15.53:0/1376961224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 0x7f48f00ffa60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f48e0009910 tx=0x7f48e0009c20 comp rx=0 tx=0).stop 2023-07-14T19:18:10.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.780+0000 7f48f602a700 1 -- 172.21.15.53:0/1376961224 shutdown_connections 2023-07-14T19:18:10.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.780+0000 7f48f602a700 1 --2- 172.21.15.53:0/1376961224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 0x7f48f00ffa60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.780+0000 7f48f602a700 1 -- 172.21.15.53:0/1376961224 >> 172.21.15.53:0/1376961224 conn(0x7f48f0075040 msgr2=0x7f48f0075440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.780+0000 7f48f602a700 1 -- 172.21.15.53:0/1376961224 shutdown_connections 2023-07-14T19:18:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.780+0000 7f48f602a700 1 -- 172.21.15.53:0/1376961224 wait complete. 2023-07-14T19:18:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.780+0000 7f48f602a700 1 Processor -- start 2023-07-14T19:18:10.783 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.780+0000 7f48f602a700 1 -- start start 2023-07-14T19:18:10.784 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48f602a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 0x7f48f0195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:10.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48f602a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f0195fb0 con 0x7f48f00ff690 2023-07-14T19:18:10.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48ef7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 0x7f48f0195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:10.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48ef7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 0x7f48f0195a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49292/0 (socket says 172.21.15.53:49292) 2023-07-14T19:18:10.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48ef7fe700 1 -- 172.21.15.53:0/1572200286 learned_addr learned my addr 172.21.15.53:0/1572200286 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:18:10.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48ef7fe700 1 -- 172.21.15.53:0/1572200286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e00095f0 con 0x7f48f00ff690 2023-07-14T19:18:10.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48ef7fe700 1 --2- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 0x7f48f0195a70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f48e00092e0 tx=0x7f48e0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:10.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48ecff9700 1 -- 172.21.15.53:0/1572200286 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48e001d070 con 0x7f48f00ff690 2023-07-14T19:18:10.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48ecff9700 1 -- 172.21.15.53:0/1572200286 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48e000bc10 con 0x7f48f00ff690 2023-07-14T19:18:10.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f01961b0 con 0x7f48f00ff690 2023-07-14T19:18:10.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48ecff9700 1 -- 172.21.15.53:0/1572200286 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48e0003930 con 0x7f48f00ff690 2023-07-14T19:18:10.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.781+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f01965d0 con 0x7f48f00ff690 2023-07-14T19:18:10.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.782+0000 7f48ecff9700 1 -- 172.21.15.53:0/1572200286 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f48e0003bb0 con 0x7f48f00ff690 2023-07-14T19:18:10.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.782+0000 7f48ecff9700 1 --2- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48d80387b0 0x7f48d803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:10.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.782+0000 7f48ecff9700 1 -- 172.21.15.53:0/1572200286 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f48e004c4e0 con 0x7f48f00ff690 2023-07-14T19:18:10.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.783+0000 7f48eeffd700 1 --2- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48d80387b0 0x7f48d803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:10.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.782+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f0103bf0 con 0x7f48f00ff690 2023-07-14T19:18:10.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.783+0000 7f48eeffd700 1 --2- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48d80387b0 0x7f48d803ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f48e4006fd0 tx=0x7f48e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:10.789 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.787+0000 7f48ecff9700 1 -- 172.21.15.53:0/1572200286 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f48e0026070 con 0x7f48f00ff690 2023-07-14T19:18:10.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:11.000 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.998+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48f00fe930 con 0x7f48d80387b0 2023-07-14T19:18:11.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:10.999+0000 7f48ecff9700 1 -- 172.21.15.53:0/1572200286 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7f48f00fe930 con 0x7f48d80387b0 2023-07-14T19:18:11.001 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:18:11.001 INFO:teuthology.orchestra.run.smithi053.stdout:[{"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}] 2023-07-14T19:18:11.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.001+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48d80387b0 msgr2=0x7f48d803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:11.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.001+0000 7f48f602a700 1 --2- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48d80387b0 0x7f48d803ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f48e4006fd0 tx=0x7f48e4006e40 comp rx=0 tx=0).stop 2023-07-14T19:18:11.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.001+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 msgr2=0x7f48f0195a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:11.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.001+0000 7f48f602a700 1 --2- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 0x7f48f0195a70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f48e00092e0 tx=0x7f48e0004ff0 comp rx=0 tx=0).stop 2023-07-14T19:18:11.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.001+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 shutdown_connections 2023-07-14T19:18:11.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.001+0000 7f48f602a700 1 --2- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48d80387b0 0x7f48d803ac60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:11.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.001+0000 7f48f602a700 1 --2- 172.21.15.53:0/1572200286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f00ff690 0x7f48f0195a70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:11.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.001+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 >> 172.21.15.53:0/1572200286 conn(0x7f48f0075040 msgr2=0x7f48f00fc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:11.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.002+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 shutdown_connections 2023-07-14T19:18:11.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:11.002+0000 7f48f602a700 1 -- 172.21.15.53:0/1572200286 wait complete. 2023-07-14T19:18:11.424 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-14T19:18:11.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd crush tunables default 2023-07-14T19:18:11.985 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:18:12.233 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:12.233 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:11 smithi053 ceph-mon[96565]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:18:12.233 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:11 smithi053 ceph-mon[96565]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:13.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.173+0000 7f9d00da1700 1 -- 172.21.15.53:0/62276417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 msgr2=0x7f9cfc105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:13.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.173+0000 7f9d00da1700 1 --2- 172.21.15.53:0/62276417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 0x7f9cfc105a40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009910 tx=0x7f9cec009c20 comp rx=0 tx=0).stop 2023-07-14T19:18:13.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.173+0000 7f9d00da1700 1 -- 172.21.15.53:0/62276417 shutdown_connections 2023-07-14T19:18:13.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.173+0000 7f9d00da1700 1 --2- 172.21.15.53:0/62276417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 0x7f9cfc105a40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:13.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.173+0000 7f9d00da1700 1 -- 172.21.15.53:0/62276417 >> 172.21.15.53:0/62276417 conn(0x7f9cfc0741e0 msgr2=0x7f9cfc0745e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:13.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.173+0000 7f9d00da1700 1 -- 172.21.15.53:0/62276417 shutdown_connections 2023-07-14T19:18:13.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.174+0000 7f9d00da1700 1 -- 172.21.15.53:0/62276417 wait complete. 2023-07-14T19:18:13.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.174+0000 7f9d00da1700 1 Processor -- start 2023-07-14T19:18:13.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.174+0000 7f9d00da1700 1 -- start start 2023-07-14T19:18:13.179 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9d00da1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 0x7f9cfc0fe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:13.179 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9d00da1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc100430 con 0x7f9cfc105670 2023-07-14T19:18:13.179 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9cfb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 0x7f9cfc0fe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:13.179 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9cfb7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 0x7f9cfc0fe4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49298/0 (socket says 172.21.15.53:49298) 2023-07-14T19:18:13.179 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9cfb7fe700 1 -- 172.21.15.53:0/1936504600 learned_addr learned my addr 172.21.15.53:0/1936504600 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:18:13.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9cfb7fe700 1 -- 172.21.15.53:0/1936504600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec0095f0 con 0x7f9cfc105670 2023-07-14T19:18:13.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9cfb7fe700 1 --2- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 0x7f9cfc0fe4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9cec004f40 tx=0x7f9cec005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:13.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9cf8ff9700 1 -- 172.21.15.53:0/1936504600 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cec01d070 con 0x7f9cfc105670 2023-07-14T19:18:13.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc0fe9f0 con 0x7f9cfc105670 2023-07-14T19:18:13.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9cf8ff9700 1 -- 172.21.15.53:0/1936504600 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cec00be20 con 0x7f9cfc105670 2023-07-14T19:18:13.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9cf8ff9700 1 -- 172.21.15.53:0/1936504600 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cec022d00 con 0x7f9cfc105670 2023-07-14T19:18:13.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.175+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc0feef0 con 0x7f9cfc105670 2023-07-14T19:18:13.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.176+0000 7f9cf8ff9700 1 -- 172.21.15.53:0/1936504600 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9cec022500 con 0x7f9cfc105670 2023-07-14T19:18:13.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.176+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc104960 con 0x7f9cfc105670 2023-07-14T19:18:13.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.176+0000 7f9cf8ff9700 1 --2- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9ce40387b0 0x7f9ce403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:13.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.177+0000 7f9cf8ff9700 1 -- 172.21.15.53:0/1936504600 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9cec04c7d0 con 0x7f9cfc105670 2023-07-14T19:18:13.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.177+0000 7f9cfaffd700 1 --2- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9ce40387b0 0x7f9ce403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:13.182 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.177+0000 7f9cfaffd700 1 --2- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9ce40387b0 0x7f9ce403ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0006fd0 tx=0x7f9cf0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:13.183 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.181+0000 7f9cf8ff9700 1 -- 172.21.15.53:0/1936504600 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9cec02a540 con 0x7f9cfc105670 2023-07-14T19:18:13.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:13.388+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9cfc061c70 con 0x7f9cfc105670 2023-07-14T19:18:13.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:13 smithi053 ceph-mon[96565]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.326+0000 7f9cf8ff9700 1 -- 172.21.15.53:0/1936504600 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9cec027070 con 0x7f9cfc105670 2023-07-14T19:18:14.331 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9ce40387b0 msgr2=0x7f9ce403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:14.332 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 --2- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9ce40387b0 0x7f9ce403ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0006fd0 tx=0x7f9cf0006e40 comp rx=0 tx=0).stop 2023-07-14T19:18:14.332 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 msgr2=0x7f9cfc0fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:14.332 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 --2- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 0x7f9cfc0fe4b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9cec004f40 tx=0x7f9cec005020 comp rx=0 tx=0).stop 2023-07-14T19:18:14.332 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 shutdown_connections 2023-07-14T19:18:14.332 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 --2- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9ce40387b0 0x7f9ce403ac60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 --2- 172.21.15.53:0/1936504600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc105670 0x7f9cfc0fe4b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 >> 172.21.15.53:0/1936504600 conn(0x7f9cfc0741e0 msgr2=0x7f9cfc106c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 shutdown_connections 2023-07-14T19:18:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:14.330+0000 7f9d00da1700 1 -- 172.21.15.53:0/1936504600 wait complete. 2023-07-14T19:18:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2023-07-14T19:18:14.670 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1936504600' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T19:18:14.671 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:14.671 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:14.671 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:14.671 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:14.671 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:18:14.671 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:14.672 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:14.672 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: Updating smithi104:/etc/ceph/ceph.conf 2023-07-14T19:18:14.672 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: Updating smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:14.672 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:14.672 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:14 smithi053 ceph-mon[96565]: Updating smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:18:14.737 INFO:tasks.cephadm:Adding mon.a on smithi053 2023-07-14T19:18:14.738 INFO:tasks.cephadm:Adding mon.b on smithi070 2023-07-14T19:18:14.738 INFO:tasks.cephadm:Adding mon.c on smithi104 2023-07-14T19:18:14.739 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch apply mon '3;smithi053:172.21.15.53=a;smithi070:172.21.15.70=b;smithi104:172.21.15.104=c' 2023-07-14T19:18:15.311 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:15.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:15 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1936504600' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T19:18:15.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:15 smithi053 ceph-mon[96565]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:15.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:15.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:15.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:15.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:15 smithi053 ceph-mon[96565]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:16.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.387+0000 7fa7e7194700 1 -- 172.21.15.104:0/2688568391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 msgr2=0x7fa7e01049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:16.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.387+0000 7fa7e7194700 1 --2- 172.21.15.104:0/2688568391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 0x7fa7e01049e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009910 tx=0x7fa7d4009c20 comp rx=0 tx=0).stop 2023-07-14T19:18:16.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.387+0000 7fa7e7194700 1 -- 172.21.15.104:0/2688568391 shutdown_connections 2023-07-14T19:18:16.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.387+0000 7fa7e7194700 1 --2- 172.21.15.104:0/2688568391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 0x7fa7e01049e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:16.389 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.387+0000 7fa7e7194700 1 -- 172.21.15.104:0/2688568391 >> 172.21.15.104:0/2688568391 conn(0x7fa7e0074310 msgr2=0x7fa7e0074710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:16.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.387+0000 7fa7e7194700 1 -- 172.21.15.104:0/2688568391 shutdown_connections 2023-07-14T19:18:16.390 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.387+0000 7fa7e7194700 1 -- 172.21.15.104:0/2688568391 wait complete. 2023-07-14T19:18:16.391 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.387+0000 7fa7e7194700 1 Processor -- start 2023-07-14T19:18:16.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.388+0000 7fa7e7194700 1 -- start start 2023-07-14T19:18:16.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.388+0000 7fa7e7194700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 0x7fa7e019ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:16.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.388+0000 7fa7e7194700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e019cfc0 con 0x7fa7e0104610 2023-07-14T19:18:16.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.388+0000 7fa7e4f30700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 0x7fa7e019ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:16.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.388+0000 7fa7e4f30700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 0x7fa7e019ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.104:49090/0 (socket says 172.21.15.104:49090) 2023-07-14T19:18:16.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.388+0000 7fa7e4f30700 1 -- 172.21.15.104:0/2700832646 learned_addr learned my addr 172.21.15.104:0/2700832646 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:18:16.392 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.388+0000 7fa7e4f30700 1 -- 172.21.15.104:0/2700832646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40095f0 con 0x7fa7e0104610 2023-07-14T19:18:16.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.389+0000 7fa7e4f30700 1 --2- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 0x7fa7e019ca80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4004f40 tx=0x7fa7d4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:16.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.389+0000 7fa7ddffb700 1 -- 172.21.15.104:0/2700832646 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7d401d070 con 0x7fa7e0104610 2023-07-14T19:18:16.393 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.389+0000 7fa7ddffb700 1 -- 172.21.15.104:0/2700832646 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7d4004370 con 0x7fa7e0104610 2023-07-14T19:18:16.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.389+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e019d220 con 0x7fa7e0104610 2023-07-14T19:18:16.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.389+0000 7fa7ddffb700 1 -- 172.21.15.104:0/2700832646 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7d40039b0 con 0x7fa7e0104610 2023-07-14T19:18:16.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.389+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e0196a70 con 0x7fa7e0104610 2023-07-14T19:18:16.394 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.390+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e0061c70 con 0x7fa7e0104610 2023-07-14T19:18:16.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.390+0000 7fa7ddffb700 1 -- 172.21.15.104:0/2700832646 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa7d4003c20 con 0x7fa7e0104610 2023-07-14T19:18:16.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.390+0000 7fa7ddffb700 1 --2- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa7d0038800 0x7fa7d003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:16.395 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.390+0000 7fa7ddffb700 1 -- 172.21.15.104:0/2700832646 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa7d404c4f0 con 0x7fa7e0104610 2023-07-14T19:18:16.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.390+0000 7fa7dffff700 1 --2- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa7d0038800 0x7fa7d003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:16.396 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.391+0000 7fa7dffff700 1 --2- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa7d0038800 0x7fa7d003acb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc006fd0 tx=0x7fa7cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:16.397 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.395+0000 7fa7ddffb700 1 -- 172.21.15.104:0/2700832646 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa7d402a980 con 0x7fa7e0104610 2023-07-14T19:18:16.588 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.586+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi070:172.21.15.70=b;smithi104:172.21.15.104=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7e0197660 con 0x7fa7d0038800 2023-07-14T19:18:16.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.594+0000 7fa7ddffb700 1 -- 172.21.15.104:0/2700832646 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa7e0197660 con 0x7fa7d0038800 2023-07-14T19:18:16.595 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mon update... 2023-07-14T19:18:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.595+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa7d0038800 msgr2=0x7fa7d003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:16.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.595+0000 7fa7e7194700 1 --2- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa7d0038800 0x7fa7d003acb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc006fd0 tx=0x7fa7cc006e40 comp rx=0 tx=0).stop 2023-07-14T19:18:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.595+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 msgr2=0x7fa7e019ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.595+0000 7fa7e7194700 1 --2- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 0x7fa7e019ca80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4004f40 tx=0x7fa7d4005020 comp rx=0 tx=0).stop 2023-07-14T19:18:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.595+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 shutdown_connections 2023-07-14T19:18:16.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.595+0000 7fa7e7194700 1 --2- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa7d0038800 0x7fa7d003acb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.595+0000 7fa7e7194700 1 --2- 172.21.15.104:0/2700832646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7e0104610 0x7fa7e019ca80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.595+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 >> 172.21.15.104:0/2700832646 conn(0x7fa7e0074310 msgr2=0x7fa7e00680d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:16.600 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.596+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 shutdown_connections 2023-07-14T19:18:16.601 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:16.596+0000 7fa7e7194700 1 -- 172.21.15.104:0/2700832646 wait complete. 2023-07-14T19:18:17.088 DEBUG:teuthology.orchestra.run.smithi070:mon.b> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.b.service 2023-07-14T19:18:17.091 DEBUG:teuthology.orchestra.run.smithi104:mon.c> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.c.service 2023-07-14T19:18:17.093 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T19:18:17.094 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T19:18:17.123 INFO:journalctl@ceph.mon.c.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:18:17.126 INFO:journalctl@ceph.mon.b.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:18:17.919 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:18.075 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi070:172.21.15.70=b;smithi104:172.21.15.104=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:18.075 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: Saving service mon spec with placement smithi053:172.21.15.53=a;smithi070:172.21.15.70=b;smithi104:172.21.15.104=c;count:3 2023-07-14T19:18:18.075 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:18.076 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:18.076 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:18.076 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:18.076 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:18.076 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:18.077 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:18.077 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: Deploying daemon mon.c on smithi104 2023-07-14T19:18:18.077 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:17 smithi053 ceph-mon[96565]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:19.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:19 smithi104 systemd[1]: Starting Ceph mon.c for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:18:19.773 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.772+0000 7f35e76a8700 1 -- 172.21.15.104:0/1923200808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 msgr2=0x7f35e0100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.772+0000 7f35e76a8700 1 --2- 172.21.15.104:0/1923200808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 0x7f35e0100cb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009910 tx=0x7f35dc009c20 comp rx=0 tx=0).stop 2023-07-14T19:18:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.772+0000 7f35e76a8700 1 -- 172.21.15.104:0/1923200808 shutdown_connections 2023-07-14T19:18:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.772+0000 7f35e76a8700 1 --2- 172.21.15.104:0/1923200808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 0x7f35e0100cb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.772+0000 7f35e76a8700 1 -- 172.21.15.104:0/1923200808 >> 172.21.15.104:0/1923200808 conn(0x7f35e00fa140 msgr2=0x7f35e00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.772+0000 7f35e76a8700 1 -- 172.21.15.104:0/1923200808 shutdown_connections 2023-07-14T19:18:19.774 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.772+0000 7f35e76a8700 1 -- 172.21.15.104:0/1923200808 wait complete. 2023-07-14T19:18:19.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.772+0000 7f35e76a8700 1 Processor -- start 2023-07-14T19:18:19.775 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.772+0000 7f35e76a8700 1 -- start start 2023-07-14T19:18:19.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.773+0000 7f35e76a8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 0x7f35e019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:19.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.773+0000 7f35e76a8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e019af20 con 0x7f35e01008e0 2023-07-14T19:18:19.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.773+0000 7f35e5444700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 0x7f35e019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:19.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.773+0000 7f35e5444700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 0x7f35e019a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.104:49096/0 (socket says 172.21.15.104:49096) 2023-07-14T19:18:19.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.773+0000 7f35e5444700 1 -- 172.21.15.104:0/3475777478 learned_addr learned my addr 172.21.15.104:0/3475777478 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:18:19.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.773+0000 7f35e5444700 1 -- 172.21.15.104:0/3475777478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc0095f0 con 0x7f35e01008e0 2023-07-14T19:18:19.776 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.773+0000 7f35e5444700 1 --2- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 0x7f35e019a9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f35dc005850 tx=0x7f35dc004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:19.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.773+0000 7f35d67fc700 1 -- 172.21.15.104:0/3475777478 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35dc01d070 con 0x7f35e01008e0 2023-07-14T19:18:19.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.774+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e019b120 con 0x7f35e01008e0 2023-07-14T19:18:19.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.774+0000 7f35d67fc700 1 -- 172.21.15.104:0/3475777478 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35dc004370 con 0x7f35e01008e0 2023-07-14T19:18:19.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.774+0000 7f35d67fc700 1 -- 172.21.15.104:0/3475777478 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35dc003ab0 con 0x7f35e01008e0 2023-07-14T19:18:19.777 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.774+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e019b540 con 0x7f35e01008e0 2023-07-14T19:18:19.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.774+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e0061c70 con 0x7f35e01008e0 2023-07-14T19:18:19.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.775+0000 7f35d67fc700 1 -- 172.21.15.104:0/3475777478 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f35dc00b6f0 con 0x7f35e01008e0 2023-07-14T19:18:19.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.775+0000 7f35d67fc700 1 --2- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35cc0387b0 0x7f35cc03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:19.778 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.775+0000 7f35d67fc700 1 -- 172.21.15.104:0/3475777478 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f35dc04c550 con 0x7f35e01008e0 2023-07-14T19:18:19.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.775+0000 7f35e4c43700 1 --2- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35cc0387b0 0x7f35cc03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:19.779 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.776+0000 7f35e4c43700 1 --2- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35cc0387b0 0x7f35cc03ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f35d0006fd0 tx=0x7f35d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:19.782 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:19.780+0000 7f35d67fc700 1 -- 172.21.15.104:0/3475777478 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f35dc02a590 con 0x7f35e01008e0 2023-07-14T19:18:19.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:19 smithi053 ceph-mon[96565]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.060+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35e0195f70 con 0x7f35e01008e0 2023-07-14T19:18:20.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.061+0000 7f35d67fc700 1 -- 172.21.15.104:0/3475777478 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f35dc026030 con 0x7f35e01008e0 2023-07-14T19:18:20.063 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:18:20.063 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","modified":"2023-07-14T19:16:45.602861Z","created":"2023-07-14T19:16:45.602861Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-14T19:18:20.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35cc0387b0 msgr2=0x7f35cc03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:20.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 --2- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35cc0387b0 0x7f35cc03ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f35d0006fd0 tx=0x7f35d0006e40 comp rx=0 tx=0).stop 2023-07-14T19:18:20.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 msgr2=0x7f35e019a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:20.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 --2- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 0x7f35e019a9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f35dc005850 tx=0x7f35dc004ff0 comp rx=0 tx=0).stop 2023-07-14T19:18:20.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 shutdown_connections 2023-07-14T19:18:20.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 --2- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35cc0387b0 0x7f35cc03ac60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 --2- 172.21.15.104:0/3475777478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35e01008e0 0x7f35e019a9e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 >> 172.21.15.104:0/3475777478 conn(0x7f35e00fa140 msgr2=0x7f35e0103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 shutdown_connections 2023-07-14T19:18:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:20.063+0000 7f35e76a8700 1 -- 172.21.15.104:0/3475777478 wait complete. 2023-07-14T19:18:20.066 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2023-07-14T19:18:20.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:19 smithi104 podman[98382]: 2023-07-14T19:18:20.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T19:18:20.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T19:18:20.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: pidfile_write: ignore empty --pid-file 2023-07-14T19:18:20.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: load: jerasure load: lrc 2023-07-14T19:18:20.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: RocksDB version: 7.9.2 2023-07-14T19:18:20.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Git sha 0 2023-07-14T19:18:20.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T19:18:20.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: DB SUMMARY 2023-07-14T19:18:20.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: DB Session ID: 3RNROAKS2V7AB1CY6NL3 2023-07-14T19:18:20.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: CURRENT file: CURRENT 2023-07-14T19:18:20.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T19:18:20.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T19:18:20.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-14T19:18:20.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-14T19:18:20.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.error_if_exists: 0 2023-07-14T19:18:20.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.create_if_missing: 0 2023-07-14T19:18:20.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.paranoid_checks: 1 2023-07-14T19:18:20.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T19:18:20.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T19:18:20.265 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T19:18:20.265 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.env: 0x56307e9a0c60 2023-07-14T19:18:20.265 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T19:18:20.265 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.info_log: 0x56307fdfd2a0 2023-07-14T19:18:20.265 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T19:18:20.265 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.statistics: (nil) 2023-07-14T19:18:20.265 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.use_fsync: 0 2023-07-14T19:18:20.265 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_log_file_size: 0 2023-07-14T19:18:20.266 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T19:18:20.266 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T19:18:20.266 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T19:18:20.266 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T19:18:20.266 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.allow_fallocate: 1 2023-07-14T19:18:20.266 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T19:18:20.266 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T19:18:20.267 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.use_direct_reads: 0 2023-07-14T19:18:20.267 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T19:18:20.267 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T19:18:20.267 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.db_log_dir: 2023-07-14T19:18:20.267 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.wal_dir: 2023-07-14T19:18:20.267 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T19:18:20.267 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T19:18:20.267 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T19:18:20.268 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T19:18:20.268 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T19:18:20.268 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T19:18:20.268 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T19:18:20.268 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T19:18:20.268 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.write_buffer_manager: 0x56307fd1e3c0 2023-07-14T19:18:20.268 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T19:18:20.268 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T19:18:20.269 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T19:18:20.269 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T19:18:20.269 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T19:18:20.269 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T19:18:20.269 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T19:18:20.269 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T19:18:20.269 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.unordered_write: 0 2023-07-14T19:18:20.270 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T19:18:20.270 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T19:18:20.270 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T19:18:20.270 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T19:18:20.270 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.row_cache: None 2023-07-14T19:18:20.270 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.wal_filter: None 2023-07-14T19:18:20.270 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T19:18:20.270 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T19:18:20.271 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.two_write_queues: 0 2023-07-14T19:18:20.271 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T19:18:20.271 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.wal_compression: 0 2023-07-14T19:18:20.271 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.atomic_flush: 0 2023-07-14T19:18:20.271 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T19:18:20.271 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T19:18:20.271 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T19:18:20.272 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.log_readahead_size: 0 2023-07-14T19:18:20.272 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T19:18:20.272 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T19:18:20.272 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T19:18:20.272 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T19:18:20.272 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T19:18:20.272 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T19:18:20.273 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T19:18:20.273 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_background_jobs: 2 2023-07-14T19:18:20.273 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_background_compactions: -1 2023-07-14T19:18:20.273 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_subcompactions: 1 2023-07-14T19:18:20.273 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T19:18:20.273 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T19:18:20.273 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T19:18:20.274 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T19:18:20.274 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T19:18:20.274 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T19:18:20.274 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T19:18:20.274 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T19:18:20.274 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_open_files: -1 2023-07-14T19:18:20.274 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T19:18:20.274 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T19:18:20.275 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T19:18:20.275 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T19:18:20.275 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_background_flushes: -1 2023-07-14T19:18:20.275 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Compression algorithms supported: 2023-07-14T19:18:20.275 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T19:18:20.275 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: kZSTD supported: 0 2023-07-14T19:18:20.275 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: kXpressCompression supported: 0 2023-07-14T19:18:20.275 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T19:18:20.276 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: kZlibCompression supported: 1 2023-07-14T19:18:20.276 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: kSnappyCompression supported: 1 2023-07-14T19:18:20.276 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: kLZ4Compression supported: 1 2023-07-14T19:18:20.276 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: kBZip2Compression supported: 0 2023-07-14T19:18:20.276 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T19:18:20.276 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T19:18:20.276 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-14T19:18:20.277 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T19:18:20.277 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T19:18:20.277 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.merge_operator: 2023-07-14T19:18:20.277 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_filter: None 2023-07-14T19:18:20.277 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T19:18:20.277 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T19:18:20.277 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T19:18:20.278 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T19:18:20.278 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56307fdfd3a0) 2023-07-14T19:18:20.278 INFO:journalctl@ceph.mon.c.smithi104.stdout: cache_index_and_filter_blocks: 1 2023-07-14T19:18:20.278 INFO:journalctl@ceph.mon.c.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T19:18:20.278 INFO:journalctl@ceph.mon.c.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T19:18:20.278 INFO:journalctl@ceph.mon.c.smithi104.stdout: pin_top_level_index_and_filter: 1 2023-07-14T19:18:20.278 INFO:journalctl@ceph.mon.c.smithi104.stdout: index_type: 0 2023-07-14T19:18:20.279 INFO:journalctl@ceph.mon.c.smithi104.stdout: data_block_index_type: 0 2023-07-14T19:18:20.279 INFO:journalctl@ceph.mon.c.smithi104.stdout: index_shortening: 1 2023-07-14T19:18:20.279 INFO:journalctl@ceph.mon.c.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T19:18:20.279 INFO:journalctl@ceph.mon.c.smithi104.stdout: checksum: 4 2023-07-14T19:18:20.279 INFO:journalctl@ceph.mon.c.smithi104.stdout: no_block_cache: 0 2023-07-14T19:18:20.279 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_cache: 0x56307fdbd1f0 2023-07-14T19:18:20.279 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_cache_name: BinnedLRUCache 2023-07-14T19:18:20.279 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_cache_options: 2023-07-14T19:18:20.280 INFO:journalctl@ceph.mon.c.smithi104.stdout: capacity : 536870912 2023-07-14T19:18:20.280 INFO:journalctl@ceph.mon.c.smithi104.stdout: num_shard_bits : 4 2023-07-14T19:18:20.280 INFO:journalctl@ceph.mon.c.smithi104.stdout: strict_capacity_limit : 0 2023-07-14T19:18:20.280 INFO:journalctl@ceph.mon.c.smithi104.stdout: high_pri_pool_ratio: 0.000 2023-07-14T19:18:20.280 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_cache_compressed: (nil) 2023-07-14T19:18:20.280 INFO:journalctl@ceph.mon.c.smithi104.stdout: persistent_cache: (nil) 2023-07-14T19:18:20.280 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_size: 4096 2023-07-14T19:18:20.281 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_size_deviation: 10 2023-07-14T19:18:20.281 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_restart_interval: 16 2023-07-14T19:18:20.281 INFO:journalctl@ceph.mon.c.smithi104.stdout: index_block_restart_interval: 1 2023-07-14T19:18:20.281 INFO:journalctl@ceph.mon.c.smithi104.stdout: metadata_block_size: 4096 2023-07-14T19:18:20.281 INFO:journalctl@ceph.mon.c.smithi104.stdout: partition_filters: 0 2023-07-14T19:18:20.281 INFO:journalctl@ceph.mon.c.smithi104.stdout: use_delta_encoding: 1 2023-07-14T19:18:20.281 INFO:journalctl@ceph.mon.c.smithi104.stdout: filter_policy: bloomfilter 2023-07-14T19:18:20.282 INFO:journalctl@ceph.mon.c.smithi104.stdout: whole_key_filtering: 1 2023-07-14T19:18:20.282 INFO:journalctl@ceph.mon.c.smithi104.stdout: verify_compression: 0 2023-07-14T19:18:20.282 INFO:journalctl@ceph.mon.c.smithi104.stdout: read_amp_bytes_per_bit: 0 2023-07-14T19:18:20.282 INFO:journalctl@ceph.mon.c.smithi104.stdout: format_version: 5 2023-07-14T19:18:20.282 INFO:journalctl@ceph.mon.c.smithi104.stdout: enable_index_compression: 1 2023-07-14T19:18:20.282 INFO:journalctl@ceph.mon.c.smithi104.stdout: block_align: 0 2023-07-14T19:18:20.282 INFO:journalctl@ceph.mon.c.smithi104.stdout: max_auto_readahead_size: 262144 2023-07-14T19:18:20.282 INFO:journalctl@ceph.mon.c.smithi104.stdout: prepopulate_block_cache: 0 2023-07-14T19:18:20.283 INFO:journalctl@ceph.mon.c.smithi104.stdout: initial_auto_readahead_size: 8192 2023-07-14T19:18:20.283 INFO:journalctl@ceph.mon.c.smithi104.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T19:18:20.283 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T19:18:20.283 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T19:18:20.283 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression: NoCompression 2023-07-14T19:18:20.283 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T19:18:20.283 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T19:18:20.284 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T19:18:20.284 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.num_levels: 7 2023-07-14T19:18:20.284 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T19:18:20.284 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T19:18:20.284 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T19:18:20.284 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T19:18:20.284 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T19:18:20.284 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T19:18:20.285 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T19:18:20.285 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:18:20.286 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T19:18:20.286 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T19:18:20.286 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:18:20.286 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:18:20.286 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T19:18:20.287 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T19:18:20.287 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T19:18:20.287 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T19:18:20.287 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:18:20.287 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:18:20.287 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T19:18:20.288 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T19:18:20.288 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:18:20.288 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T19:18:20.288 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T19:18:20.288 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T19:18:20.288 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T19:18:20.288 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T19:18:20.288 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T19:18:20.289 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T19:18:20.289 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T19:18:20.289 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T19:18:20.289 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T19:18:20.289 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T19:18:20.289 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T19:18:20.289 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T19:18:20.290 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T19:18:20.290 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T19:18:20.290 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T19:18:20.290 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T19:18:20.290 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T19:18:20.290 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T19:18:20.290 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T19:18:20.291 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T19:18:20.291 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T19:18:20.291 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T19:18:20.291 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T19:18:20.291 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T19:18:20.291 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T19:18:20.291 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T19:18:20.292 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T19:18:20.292 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T19:18:20.292 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T19:18:20.292 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T19:18:20.292 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T19:18:20.292 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.table_properties_collectors: 2023-07-14T19:18:20.292 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.inplace_update_support: 0 2023-07-14T19:18:20.292 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T19:18:20.293 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T19:18:20.293 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T19:18:20.293 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T19:18:20.293 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.bloom_locality: 0 2023-07-14T19:18:20.293 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.max_successive_merges: 0 2023-07-14T19:18:20.293 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T19:18:20.293 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T19:18:20.294 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T19:18:20.294 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T19:18:20.294 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.ttl: 2592000 2023-07-14T19:18:20.294 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T19:18:20.294 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T19:18:20.294 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T19:18:20.294 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.enable_blob_files: false 2023-07-14T19:18:20.294 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.min_blob_size: 0 2023-07-14T19:18:20.295 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T19:18:20.295 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T19:18:20.295 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T19:18:20.295 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T19:18:20.295 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T19:18:20.295 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T19:18:20.295 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T19:18:20.295 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T19:18:20.296 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T19:18:20.296 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T19:18:20.296 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c5977d9b-f893-432f-964f-85981533085d 2023-07-14T19:18:20.296 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362300151880, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T19:18:20.296 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T19:18:20.296 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362300152283, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689362300, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c5977d9b-f893-432f-964f-85981533085d", "db_session_id": "3RNROAKS2V7AB1CY6NL3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T19:18:20.296 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362300152350, "job": 1, "event": "recovery_finished"} 2023-07-14T19:18:20.297 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T19:18:20.297 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T19:18:20.297 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56307fe56000 2023-07-14T19:18:20.297 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: DB pointer 0x56307fe42000 2023-07-14T19:18:20.297 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T19:18:20.297 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T19:18:20.297 INFO:journalctl@ceph.mon.c.smithi104.stdout: ** DB Stats ** 2023-07-14T19:18:20.297 INFO:journalctl@ceph.mon.c.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:18:20.298 INFO:journalctl@ceph.mon.c.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T19:18:20.298 INFO:journalctl@ceph.mon.c.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:18:20.298 INFO:journalctl@ceph.mon.c.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:18:20.298 INFO:journalctl@ceph.mon.c.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T19:18:20.298 INFO:journalctl@ceph.mon.c.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:18:20.298 INFO:journalctl@ceph.mon.c.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:18:20.299 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2023-07-14T19:18:20.299 INFO:journalctl@ceph.mon.c.smithi104.stdout: ** Compaction Stats [default] ** 2023-07-14T19:18:20.299 INFO:journalctl@ceph.mon.c.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:18:20.299 INFO:journalctl@ceph.mon.c.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T19:18:20.299 INFO:journalctl@ceph.mon.c.smithi104.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:18:20.300 INFO:journalctl@ceph.mon.c.smithi104.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:18:20.300 INFO:journalctl@ceph.mon.c.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:18:20.300 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2023-07-14T19:18:20.300 INFO:journalctl@ceph.mon.c.smithi104.stdout: ** Compaction Stats [default] ** 2023-07-14T19:18:20.300 INFO:journalctl@ceph.mon.c.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:18:20.300 INFO:journalctl@ceph.mon.c.smithi104.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T19:18:20.300 INFO:journalctl@ceph.mon.c.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:18:20.300 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2023-07-14T19:18:20.301 INFO:journalctl@ceph.mon.c.smithi104.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T19:18:20.301 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2023-07-14T19:18:20.301 INFO:journalctl@ceph.mon.c.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:18:20.301 INFO:journalctl@ceph.mon.c.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T19:18:20.301 INFO:journalctl@ceph.mon.c.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T19:18:20.301 INFO:journalctl@ceph.mon.c.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T19:18:20.301 INFO:journalctl@ceph.mon.c.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T19:18:20.302 INFO:journalctl@ceph.mon.c.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T19:18:20.302 INFO:journalctl@ceph.mon.c.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:18:20.302 INFO:journalctl@ceph.mon.c.smithi104.stdout: Interval compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:18:20.302 INFO:journalctl@ceph.mon.c.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T19:18:20.302 INFO:journalctl@ceph.mon.c.smithi104.stdout: Block cache BinnedLRUCache@0x56307fdbd1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2023-07-14T19:18:20.302 INFO:journalctl@ceph.mon.c.smithi104.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T19:18:20.302 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2023-07-14T19:18:20.302 INFO:journalctl@ceph.mon.c.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T19:18:20.303 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-14T19:18:20.303 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: using public_addr v2:172.21.15.104:0/0 -> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2023-07-14T19:18:20.303 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: starting mon.c rank -1 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:18:20.303 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(???) e0 preinit fsid c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:18:20.303 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).mds e1 new map 2023-07-14T19:18:20.303 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-14T19:18:20.303 INFO:journalctl@ceph.mon.c.smithi104.stdout: e1 2023-07-14T19:18:20.304 INFO:journalctl@ceph.mon.c.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:18:20.304 INFO:journalctl@ceph.mon.c.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:18:20.304 INFO:journalctl@ceph.mon.c.smithi104.stdout: legacy client fscid: -1 2023-07-14T19:18:20.304 INFO:journalctl@ceph.mon.c.smithi104.stdout: 2023-07-14T19:18:20.304 INFO:journalctl@ceph.mon.c.smithi104.stdout: No filesystems configured 2023-07-14T19:18:20.304 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T19:18:20.304 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T19:18:20.305 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T19:18:20.305 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T19:18:20.305 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T19:18:20.305 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T19:18:20.305 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T19:18:20.305 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:18:20.305 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:18:20.305 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:18:20.306 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:20.306 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.306 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.306 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.306 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.306 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:18:20.306 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:20.306 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:20.307 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Updating smithi070:/etc/ceph/ceph.conf 2023-07-14T19:18:20.307 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Deploying cephadm binary to smithi104 2023-07-14T19:18:20.307 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Updating smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:20.307 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:20.307 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Updating smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:18:20.307 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:20.307 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Added host smithi104 2023-07-14T19:18:20.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:20.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:20.308 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:18:20.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:20.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:20.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1936504600' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T19:18:20.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.309 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.310 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.310 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:18:20.310 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:20.310 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:20.310 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Updating smithi104:/etc/ceph/ceph.conf 2023-07-14T19:18:20.310 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Updating smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:20.310 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:20.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Updating smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:18:20.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1936504600' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T19:18:20.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:20.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:20.311 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi070:172.21.15.70=b;smithi104:172.21.15.104=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:20.312 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Saving service mon spec with placement smithi053:172.21.15.53=a;smithi070:172.21.15.70=b;smithi104:172.21.15.104=c;count:3 2023-07-14T19:18:20.312 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.312 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:20.312 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:20.312 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:20.312 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:20.312 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:20.313 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:20.313 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: Deploying daemon mon.c on smithi104 2023-07-14T19:18:20.313 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:20.313 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:20.313 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T19:18:20.313 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-14T19:18:20.313 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta expand map: {default=false} 2023-07-14T19:18:20.313 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta from 'false' to 'false' 2023-07-14T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta expanded map: {default=false} 2023-07-14T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta expand map: {default=info} 2023-07-14T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta from 'info' to 'info' 2023-07-14T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta expanded map: {default=info} 2023-07-14T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta expand map: {default=daemon} 2023-07-14T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T19:18:20.315 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta expand map: {default=debug} 2023-07-14T19:18:20.315 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T19:18:20.315 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 ceph-mon[98435]: expand_channel_meta expanded map: {default=debug} 2023-07-14T19:18:20.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 bash[98382]: 6c421be06c2c2cef75569f5e4487bb0ee5b3c5f1ad4c9afc9be16a9b91ba3153 2023-07-14T19:18:20.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:20 smithi104 systemd[1]: Started Ceph mon.c for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:18:21.606 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T19:18:21.607 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T19:18:22.168 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:18:25.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:18:25.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: mon.a calling monitor election 2023-07-14T19:18:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: mon.c calling monitor election 2023-07-14T19:18:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:18:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: monmap e2: 2 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],c=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} 2023-07-14T19:18:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: fsmap 2023-07-14T19:18:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: mgrmap e13: x(active, since 43s) 2023-07-14T19:18:25.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: overall HEALTH_OK 2023-07-14T19:18:25.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:25.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:25.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:25.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:25.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:25.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:18:25.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: mon.a calling monitor election 2023-07-14T19:18:25.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:25.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: mon.c calling monitor election 2023-07-14T19:18:25.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:25.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:25.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:18:25.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: monmap e2: 2 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],c=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} 2023-07-14T19:18:25.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: fsmap 2023-07-14T19:18:25.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:25.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: mgrmap e13: x(active, since 43s) 2023-07-14T19:18:25.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: overall HEALTH_OK 2023-07-14T19:18:25.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:25.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:25.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:25.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:25.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:26.398 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.396+0000 7ff704b45700 1 -- 172.21.15.104:0/2276420830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e80035c0 msgr2=0x7ff6e8005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:26.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.396+0000 7ff704b45700 1 --2- 172.21.15.104:0/2276420830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e80035c0 0x7ff6e8005a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff7000ffc70 tx=0x7ff6f400fe60 comp rx=0 tx=0).stop 2023-07-14T19:18:26.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.397+0000 7ff704b45700 1 -- 172.21.15.104:0/2276420830 shutdown_connections 2023-07-14T19:18:26.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.397+0000 7ff704b45700 1 --2- 172.21.15.104:0/2276420830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e80035c0 0x7ff6e8005a40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:26.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.397+0000 7ff704b45700 1 -- 172.21.15.104:0/2276420830 >> 172.21.15.104:0/2276420830 conn(0x7ff700074500 msgr2=0x7ff700074900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:26.399 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.397+0000 7ff704b45700 1 -- 172.21.15.104:0/2276420830 shutdown_connections 2023-07-14T19:18:26.400 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.397+0000 7ff704b45700 1 -- 172.21.15.104:0/2276420830 wait complete. 2023-07-14T19:18:26.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.398+0000 7ff704b45700 1 Processor -- start 2023-07-14T19:18:26.401 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.398+0000 7ff704b45700 1 -- start start 2023-07-14T19:18:26.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.398+0000 7ff704b45700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e80035c0 0x7ff700195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:26.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff704b45700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 0x7ff7001964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:26.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff704b45700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70019eba0 con 0x7ff6e80035c0 2023-07-14T19:18:26.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff704b45700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70019ece0 con 0x7ff700196070 2023-07-14T19:18:26.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fdd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 0x7ff7001964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:26.402 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fdd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 0x7ff7001964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56836/0 (socket says 172.21.15.104:56836) 2023-07-14T19:18:26.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fdd9b700 1 -- 172.21.15.104:0/1267045183 learned_addr learned my addr 172.21.15.104:0/1267045183 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:18:26.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fe59c700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e80035c0 0x7ff700195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:26.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fdd9b700 1 -- 172.21.15.104:0/1267045183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 msgr2=0x7ff7001964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T19:18:26.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fdd9b700 1 -- 172.21.15.104:0/1267045183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 msgr2=0x7ff7001964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T19:18:26.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fdd9b700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 0x7ff7001964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T19:18:26.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fdd9b700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 0x7ff7001964e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:18:26.403 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fe59c700 1 -- 172.21.15.104:0/1267045183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 msgr2=0x7ff7001964e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:18:26.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fe59c700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 0x7ff7001964e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:26.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fe59c700 1 -- 172.21.15.104:0/1267045183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f400f8b0 con 0x7ff6e80035c0 2023-07-14T19:18:26.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.399+0000 7ff6fe59c700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e80035c0 0x7ff700195b30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40046a0 tx=0x7ff6f4013c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:26.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.400+0000 7ff6ef7fe700 1 -- 172.21.15.104:0/1267045183 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7ff6f4023070 con 0x7ff6e80035c0 2023-07-14T19:18:26.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.400+0000 7ff6ef7fe700 1 -- 172.21.15.104:0/1267045183 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6f4004040 con 0x7ff6e80035c0 2023-07-14T19:18:26.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.400+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70019ee80 con 0x7ff6e80035c0 2023-07-14T19:18:26.404 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.400+0000 7ff6ef7fe700 1 -- 172.21.15.104:0/1267045183 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7ff6f401cb50 con 0x7ff6e80035c0 2023-07-14T19:18:26.405 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.400+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70019f3c0 con 0x7ff6e80035c0 2023-07-14T19:18:26.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.401+0000 7ff6ef7fe700 1 -- 172.21.15.104:0/1267045183 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7ff6f4031440 con 0x7ff6e80035c0 2023-07-14T19:18:26.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.401+0000 7ff6ef7fe700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6e00385f0 0x7ff6e003aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:26.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.401+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff700061c70 con 0x7ff6e80035c0 2023-07-14T19:18:26.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.401+0000 7ff6ef7fe700 1 -- 172.21.15.104:0/1267045183 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff6f404d400 con 0x7ff6e80035c0 2023-07-14T19:18:26.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.401+0000 7ff6fdd9b700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6e00385f0 0x7ff6e003aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:26.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.402+0000 7ff6fdd9b700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6e00385f0 0x7ff6e003aaa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009180 tx=0x7ff6f0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:26.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.406+0000 7ff6ef7fe700 1 -- 172.21.15.104:0/1267045183 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff6f402f950 con 0x7ff6e80035c0 2023-07-14T19:18:26.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:26 smithi053 ceph-mon[96565]: Deploying daemon mon.b on smithi070 2023-07-14T19:18:26.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:26 smithi053 ceph-mon[96565]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:26.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:26 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:26.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.666+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7000683b0 con 0x7ff6e80035c0 2023-07-14T19:18:26.668 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.667+0000 7ff6ef7fe700 1 -- 172.21.15.104:0/1267045183 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7ff6f4016150 con 0x7ff6e80035c0 2023-07-14T19:18:26.669 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:18:26.669 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","modified":"2023-07-14T19:18:20.177497Z","created":"2023-07-14T19:16:45.602861Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T19:18:26.670 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6e00385f0 msgr2=0x7ff6e003aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:26.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6e00385f0 0x7ff6e003aaa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009180 tx=0x7ff6f0008040 comp rx=0 tx=0).stop 2023-07-14T19:18:26.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e80035c0 msgr2=0x7ff700195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:26.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e80035c0 0x7ff700195b30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40046a0 tx=0x7ff6f4013c90 comp rx=0 tx=0).stop 2023-07-14T19:18:26.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 shutdown_connections 2023-07-14T19:18:26.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6e00385f0 0x7ff6e003aaa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:26.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6e80035c0 0x7ff700195b30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:26.671 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 --2- 172.21.15.104:0/1267045183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff700196070 0x7ff7001964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 >> 172.21.15.104:0/1267045183 conn(0x7ff700074500 msgr2=0x7ff700102330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 shutdown_connections 2023-07-14T19:18:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:26.669+0000 7ff704b45700 1 -- 172.21.15.104:0/1267045183 wait complete. 2023-07-14T19:18:26.672 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 2 2023-07-14T19:18:26.691 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:26 smithi104 ceph-mon[98435]: Deploying daemon mon.b on smithi070 2023-07-14T19:18:26.691 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:26 smithi104 ceph-mon[98435]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:26.691 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:26 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:27.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:27 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/1267045183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:18:27.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:27 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/1267045183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:18:28.017 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T19:18:28.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T19:18:28.137 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:27 smithi070 systemd[1]: Starting Ceph mon.b for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:18:28.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 podman[97975]: 2023-07-14T19:18:28.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 bash[97975]: 1eface503eae137aa2368ab250b1e31543e096955d2e25aeb0e0b68216cb35a0 2023-07-14T19:18:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T19:18:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T19:18:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pidfile_write: ignore empty --pid-file 2023-07-14T19:18:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: load: jerasure load: lrc 2023-07-14T19:18:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: RocksDB version: 7.9.2 2023-07-14T19:18:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Git sha 0 2023-07-14T19:18:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T19:18:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: DB SUMMARY 2023-07-14T19:18:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: DB Session ID: HV0QD2YKK6J62Z1BC05E 2023-07-14T19:18:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: CURRENT file: CURRENT 2023-07-14T19:18:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T19:18:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T19:18:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-14T19:18:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-14T19:18:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.error_if_exists: 0 2023-07-14T19:18:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.create_if_missing: 0 2023-07-14T19:18:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.paranoid_checks: 1 2023-07-14T19:18:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T19:18:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T19:18:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T19:18:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.env: 0x5572505e0c60 2023-07-14T19:18:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T19:18:28.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.info_log: 0x5572524032a0 2023-07-14T19:18:28.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T19:18:28.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.statistics: (nil) 2023-07-14T19:18:28.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.use_fsync: 0 2023-07-14T19:18:28.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_log_file_size: 0 2023-07-14T19:18:28.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T19:18:28.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T19:18:28.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T19:18:28.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T19:18:28.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.allow_fallocate: 1 2023-07-14T19:18:28.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T19:18:28.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T19:18:28.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.use_direct_reads: 0 2023-07-14T19:18:28.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T19:18:28.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T19:18:28.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.db_log_dir: 2023-07-14T19:18:28.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.wal_dir: 2023-07-14T19:18:28.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T19:18:28.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T19:18:28.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T19:18:28.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T19:18:28.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T19:18:28.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T19:18:28.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T19:18:28.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T19:18:28.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.write_buffer_manager: 0x5572523243c0 2023-07-14T19:18:28.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T19:18:28.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T19:18:28.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T19:18:28.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T19:18:28.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T19:18:28.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T19:18:28.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T19:18:28.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T19:18:28.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.unordered_write: 0 2023-07-14T19:18:28.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T19:18:28.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T19:18:28.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T19:18:28.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T19:18:28.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.row_cache: None 2023-07-14T19:18:28.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.wal_filter: None 2023-07-14T19:18:28.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T19:18:28.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T19:18:28.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.two_write_queues: 0 2023-07-14T19:18:28.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T19:18:28.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.wal_compression: 0 2023-07-14T19:18:28.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.atomic_flush: 0 2023-07-14T19:18:28.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T19:18:28.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T19:18:28.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T19:18:28.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.log_readahead_size: 0 2023-07-14T19:18:28.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T19:18:28.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T19:18:28.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T19:18:28.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T19:18:28.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T19:18:28.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T19:18:28.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T19:18:28.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_background_jobs: 2 2023-07-14T19:18:28.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_background_compactions: -1 2023-07-14T19:18:28.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_subcompactions: 1 2023-07-14T19:18:28.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T19:18:28.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T19:18:28.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T19:18:28.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T19:18:28.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T19:18:28.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T19:18:28.492 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T19:18:28.492 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T19:18:28.492 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_open_files: -1 2023-07-14T19:18:28.492 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T19:18:28.492 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T19:18:28.492 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T19:18:28.492 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T19:18:28.493 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_background_flushes: -1 2023-07-14T19:18:28.493 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Compression algorithms supported: 2023-07-14T19:18:28.493 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T19:18:28.493 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: kZSTD supported: 0 2023-07-14T19:18:28.493 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: kXpressCompression supported: 0 2023-07-14T19:18:28.493 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T19:18:28.493 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: kZlibCompression supported: 1 2023-07-14T19:18:28.494 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: kSnappyCompression supported: 1 2023-07-14T19:18:28.494 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: kLZ4Compression supported: 1 2023-07-14T19:18:28.494 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: kBZip2Compression supported: 0 2023-07-14T19:18:28.494 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T19:18:28.494 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T19:18:28.494 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-14T19:18:28.494 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T19:18:28.494 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T19:18:28.495 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.merge_operator: 2023-07-14T19:18:28.495 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_filter: None 2023-07-14T19:18:28.495 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T19:18:28.495 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T19:18:28.495 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T19:18:28.495 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T19:18:28.495 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5572524033a0) 2023-07-14T19:18:28.495 INFO:journalctl@ceph.mon.b.smithi070.stdout: cache_index_and_filter_blocks: 1 2023-07-14T19:18:28.496 INFO:journalctl@ceph.mon.b.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T19:18:28.496 INFO:journalctl@ceph.mon.b.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T19:18:28.496 INFO:journalctl@ceph.mon.b.smithi070.stdout: pin_top_level_index_and_filter: 1 2023-07-14T19:18:28.496 INFO:journalctl@ceph.mon.b.smithi070.stdout: index_type: 0 2023-07-14T19:18:28.496 INFO:journalctl@ceph.mon.b.smithi070.stdout: data_block_index_type: 0 2023-07-14T19:18:28.496 INFO:journalctl@ceph.mon.b.smithi070.stdout: index_shortening: 1 2023-07-14T19:18:28.496 INFO:journalctl@ceph.mon.b.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T19:18:28.497 INFO:journalctl@ceph.mon.b.smithi070.stdout: checksum: 4 2023-07-14T19:18:28.497 INFO:journalctl@ceph.mon.b.smithi070.stdout: no_block_cache: 0 2023-07-14T19:18:28.497 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_cache: 0x5572523c31f0 2023-07-14T19:18:28.497 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_cache_name: BinnedLRUCache 2023-07-14T19:18:28.497 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_cache_options: 2023-07-14T19:18:28.497 INFO:journalctl@ceph.mon.b.smithi070.stdout: capacity : 536870912 2023-07-14T19:18:28.497 INFO:journalctl@ceph.mon.b.smithi070.stdout: num_shard_bits : 4 2023-07-14T19:18:28.497 INFO:journalctl@ceph.mon.b.smithi070.stdout: strict_capacity_limit : 0 2023-07-14T19:18:28.498 INFO:journalctl@ceph.mon.b.smithi070.stdout: high_pri_pool_ratio: 0.000 2023-07-14T19:18:28.498 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_cache_compressed: (nil) 2023-07-14T19:18:28.498 INFO:journalctl@ceph.mon.b.smithi070.stdout: persistent_cache: (nil) 2023-07-14T19:18:28.498 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_size: 4096 2023-07-14T19:18:28.498 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_size_deviation: 10 2023-07-14T19:18:28.498 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_restart_interval: 16 2023-07-14T19:18:28.498 INFO:journalctl@ceph.mon.b.smithi070.stdout: index_block_restart_interval: 1 2023-07-14T19:18:28.498 INFO:journalctl@ceph.mon.b.smithi070.stdout: metadata_block_size: 4096 2023-07-14T19:18:28.499 INFO:journalctl@ceph.mon.b.smithi070.stdout: partition_filters: 0 2023-07-14T19:18:28.499 INFO:journalctl@ceph.mon.b.smithi070.stdout: use_delta_encoding: 1 2023-07-14T19:18:28.499 INFO:journalctl@ceph.mon.b.smithi070.stdout: filter_policy: bloomfilter 2023-07-14T19:18:28.499 INFO:journalctl@ceph.mon.b.smithi070.stdout: whole_key_filtering: 1 2023-07-14T19:18:28.499 INFO:journalctl@ceph.mon.b.smithi070.stdout: verify_compression: 0 2023-07-14T19:18:28.499 INFO:journalctl@ceph.mon.b.smithi070.stdout: read_amp_bytes_per_bit: 0 2023-07-14T19:18:28.499 INFO:journalctl@ceph.mon.b.smithi070.stdout: format_version: 5 2023-07-14T19:18:28.499 INFO:journalctl@ceph.mon.b.smithi070.stdout: enable_index_compression: 1 2023-07-14T19:18:28.500 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_align: 0 2023-07-14T19:18:28.500 INFO:journalctl@ceph.mon.b.smithi070.stdout: max_auto_readahead_size: 262144 2023-07-14T19:18:28.500 INFO:journalctl@ceph.mon.b.smithi070.stdout: prepopulate_block_cache: 0 2023-07-14T19:18:28.500 INFO:journalctl@ceph.mon.b.smithi070.stdout: initial_auto_readahead_size: 8192 2023-07-14T19:18:28.500 INFO:journalctl@ceph.mon.b.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T19:18:28.500 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T19:18:28.500 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T19:18:28.500 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression: NoCompression 2023-07-14T19:18:28.501 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T19:18:28.501 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T19:18:28.501 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T19:18:28.501 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.num_levels: 7 2023-07-14T19:18:28.501 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T19:18:28.501 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T19:18:28.501 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T19:18:28.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T19:18:28.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T19:18:28.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T19:18:28.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T19:18:28.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:18:28.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T19:18:28.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T19:18:28.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:18:28.503 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:18:28.503 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T19:18:28.503 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T19:18:28.503 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T19:18:28.503 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T19:18:28.503 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:18:28.503 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:18:28.504 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T19:18:28.504 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T19:18:28.504 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:18:28.504 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T19:18:28.504 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T19:18:28.504 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T19:18:28.504 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T19:18:28.504 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T19:18:28.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T19:18:28.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T19:18:28.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T19:18:28.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T19:18:28.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T19:18:28.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T19:18:28.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T19:18:28.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T19:18:28.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T19:18:28.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T19:18:28.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T19:18:28.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T19:18:28.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T19:18:28.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T19:18:28.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T19:18:28.507 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T19:18:28.507 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T19:18:28.507 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T19:18:28.507 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T19:18:28.507 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T19:18:28.507 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T19:18:28.507 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T19:18:28.507 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T19:18:28.508 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T19:18:28.508 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T19:18:28.508 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T19:18:28.508 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T19:18:28.508 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.table_properties_collectors: 2023-07-14T19:18:28.508 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.inplace_update_support: 0 2023-07-14T19:18:28.508 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T19:18:28.509 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T19:18:28.509 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T19:18:28.509 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T19:18:28.509 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.bloom_locality: 0 2023-07-14T19:18:28.509 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.max_successive_merges: 0 2023-07-14T19:18:28.509 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T19:18:28.509 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T19:18:28.509 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T19:18:28.510 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T19:18:28.510 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.ttl: 2592000 2023-07-14T19:18:28.510 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T19:18:28.510 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T19:18:28.510 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T19:18:28.510 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.enable_blob_files: false 2023-07-14T19:18:28.510 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.min_blob_size: 0 2023-07-14T19:18:28.510 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T19:18:28.511 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T19:18:28.511 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T19:18:28.511 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T19:18:28.511 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T19:18:28.511 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T19:18:28.511 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T19:18:28.511 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T19:18:28.511 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T19:18:28.512 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T19:18:28.512 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 36bb16c9-6d66-44d5-a7c0-ca162894db50 2023-07-14T19:18:28.512 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362308341962, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T19:18:28.512 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T19:18:28.512 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362308342367, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689362308, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "36bb16c9-6d66-44d5-a7c0-ca162894db50", "db_session_id": "HV0QD2YKK6J62Z1BC05E", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T19:18:28.512 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362308342432, "job": 1, "event": "recovery_finished"} 2023-07-14T19:18:28.512 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T19:18:28.513 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T19:18:28.513 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55725245c000 2023-07-14T19:18:28.513 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: DB pointer 0x557252448000 2023-07-14T19:18:28.513 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T19:18:28.513 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T19:18:28.513 INFO:journalctl@ceph.mon.b.smithi070.stdout: ** DB Stats ** 2023-07-14T19:18:28.513 INFO:journalctl@ceph.mon.b.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:18:28.513 INFO:journalctl@ceph.mon.b.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T19:18:28.514 INFO:journalctl@ceph.mon.b.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:18:28.514 INFO:journalctl@ceph.mon.b.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:18:28.514 INFO:journalctl@ceph.mon.b.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T19:18:28.514 INFO:journalctl@ceph.mon.b.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:18:28.514 INFO:journalctl@ceph.mon.b.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:18:28.514 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2023-07-14T19:18:28.514 INFO:journalctl@ceph.mon.b.smithi070.stdout: ** Compaction Stats [default] ** 2023-07-14T19:18:28.515 INFO:journalctl@ceph.mon.b.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:18:28.515 INFO:journalctl@ceph.mon.b.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T19:18:28.515 INFO:journalctl@ceph.mon.b.smithi070.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:18:28.515 INFO:journalctl@ceph.mon.b.smithi070.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:18:28.515 INFO:journalctl@ceph.mon.b.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:18:28.515 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2023-07-14T19:18:28.515 INFO:journalctl@ceph.mon.b.smithi070.stdout: ** Compaction Stats [default] ** 2023-07-14T19:18:28.515 INFO:journalctl@ceph.mon.b.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:18:28.516 INFO:journalctl@ceph.mon.b.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T19:18:28.516 INFO:journalctl@ceph.mon.b.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:18:28.516 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2023-07-14T19:18:28.516 INFO:journalctl@ceph.mon.b.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T19:18:28.516 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2023-07-14T19:18:28.516 INFO:journalctl@ceph.mon.b.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:18:28.516 INFO:journalctl@ceph.mon.b.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T19:18:28.516 INFO:journalctl@ceph.mon.b.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T19:18:28.517 INFO:journalctl@ceph.mon.b.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T19:18:28.517 INFO:journalctl@ceph.mon.b.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T19:18:28.517 INFO:journalctl@ceph.mon.b.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T19:18:28.517 INFO:journalctl@ceph.mon.b.smithi070.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:18:28.517 INFO:journalctl@ceph.mon.b.smithi070.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:18:28.517 INFO:journalctl@ceph.mon.b.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T19:18:28.517 INFO:journalctl@ceph.mon.b.smithi070.stdout: Block cache BinnedLRUCache@0x5572523c31f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-14T19:18:28.518 INFO:journalctl@ceph.mon.b.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T19:18:28.518 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2023-07-14T19:18:28.518 INFO:journalctl@ceph.mon.b.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T19:18:28.518 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-14T19:18:28.518 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: using public_addr v2:172.21.15.70:0/0 -> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2023-07-14T19:18:28.518 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: starting mon.b rank -1 at public addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] at bind addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:18:28.518 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(???) e0 preinit fsid c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:18:28.518 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 systemd[1]: Started Ceph mon.b for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:18:28.519 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).mds e1 new map 2023-07-14T19:18:28.519 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-14T19:18:28.519 INFO:journalctl@ceph.mon.b.smithi070.stdout: e1 2023-07-14T19:18:28.519 INFO:journalctl@ceph.mon.b.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:18:28.519 INFO:journalctl@ceph.mon.b.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:18:28.519 INFO:journalctl@ceph.mon.b.smithi070.stdout: legacy client fscid: -1 2023-07-14T19:18:28.519 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2023-07-14T19:18:28.519 INFO:journalctl@ceph.mon.b.smithi070.stdout: No filesystems configured 2023-07-14T19:18:28.520 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T19:18:28.520 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T19:18:28.520 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T19:18:28.520 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T19:18:28.520 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T19:18:28.520 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T19:18:28.520 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T19:18:28.521 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:18:28.521 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:18:28.521 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:18:28.521 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:28.521 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.521 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.521 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.521 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.522 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:18:28.522 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:28.522 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:28.522 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Updating smithi070:/etc/ceph/ceph.conf 2023-07-14T19:18:28.522 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Deploying cephadm binary to smithi104 2023-07-14T19:18:28.522 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Updating smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:28.522 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:28.522 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Updating smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:18:28.523 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.523 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.523 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.523 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.523 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.523 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Added host smithi104 2023-07-14T19:18:28.523 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:28.523 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.524 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.524 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.524 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.524 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:18:28.524 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.524 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.524 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1936504600' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T19:18:28.525 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.525 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.525 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.525 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.525 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:18:28.525 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:28.525 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:28.525 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Updating smithi104:/etc/ceph/ceph.conf 2023-07-14T19:18:28.526 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Updating smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:28.526 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:28.526 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Updating smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:18:28.526 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1936504600' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T19:18:28.526 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:28.526 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.526 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.527 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.527 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.527 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi070:172.21.15.70=b;smithi104:172.21.15.104=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:28.527 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Saving service mon spec with placement smithi053:172.21.15.53=a;smithi070:172.21.15.70=b;smithi104:172.21.15.104=c;count:3 2023-07-14T19:18:28.527 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.527 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:28.527 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:28.527 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:28.528 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.528 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:28.528 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:28.528 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Deploying daemon mon.c on smithi104 2023-07-14T19:18:28.528 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.528 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.528 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:18:28.529 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:28.529 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.a calling monitor election 2023-07-14T19:18:28.529 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:28.529 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.529 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:28.529 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.c calling monitor election 2023-07-14T19:18:28.529 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:28.529 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.530 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:28.530 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:28.530 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:18:28.530 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: monmap e2: 2 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],c=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} 2023-07-14T19:18:28.530 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: fsmap 2023-07-14T19:18:28.530 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:28.530 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mgrmap e13: x(active, since 43s) 2023-07-14T19:18:28.530 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: overall HEALTH_OK 2023-07-14T19:18:28.531 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.531 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.531 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:28.531 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:28.531 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:28.531 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: Deploying daemon mon.b on smithi070 2023-07-14T19:18:28.531 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.531 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:28.532 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/1267045183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:18:28.532 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.532 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T19:18:28.532 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta expand map: {default=false} 2023-07-14T19:18:28.532 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta from 'false' to 'false' 2023-07-14T19:18:28.532 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta expanded map: {default=false} 2023-07-14T19:18:28.532 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta expand map: {default=info} 2023-07-14T19:18:28.533 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta from 'info' to 'info' 2023-07-14T19:18:28.533 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta expanded map: {default=info} 2023-07-14T19:18:28.533 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta expand map: {default=daemon} 2023-07-14T19:18:28.533 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T19:18:28.533 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T19:18:28.533 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta expand map: {default=debug} 2023-07-14T19:18:28.533 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T19:18:28.533 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: expand_channel_meta expanded map: {default=debug} 2023-07-14T19:18:28.534 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:28 smithi070 ceph-mon[98015]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-14T19:18:28.534 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:28 smithi104 ceph-mon[98435]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:28 smithi053 ceph-mon[96565]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:28.581 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:18:33.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:18:33.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: mon.a calling monitor election 2023-07-14T19:18:33.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: mon.c calling monitor election 2023-07-14T19:18:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:33.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:18:33.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],c=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} 2023-07-14T19:18:33.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: fsmap 2023-07-14T19:18:33.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:33.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: mgrmap e13: x(active, since 52s) 2023-07-14T19:18:33.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T19:18:33.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T19:18:33.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T19:18:33.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: mon.b (rank 2) addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] is down (out of quorum) 2023-07-14T19:18:33.765 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:33.765 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:33.765 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:33.765 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:33.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:18:33.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: mon.a calling monitor election 2023-07-14T19:18:33.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: mon.c calling monitor election 2023-07-14T19:18:33.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:33.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:33.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:33.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:33.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:33.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:18:33.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],c=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} 2023-07-14T19:18:33.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: fsmap 2023-07-14T19:18:33.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:33.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: mgrmap e13: x(active, since 52s) 2023-07-14T19:18:33.825 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T19:18:33.825 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T19:18:33.825 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T19:18:33.825 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: mon.b (rank 2) addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] is down (out of quorum) 2023-07-14T19:18:33.825 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:33.825 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:33.825 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:33.825 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:34.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.156+0000 7fc13e5ef700 1 -- 172.21.15.104:0/3717892631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1200057d0 msgr2=0x7fc120007c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:34.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.156+0000 7fc13e5ef700 1 --2- 172.21.15.104:0/3717892631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1200057d0 0x7fc120007c50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc1380fe6a0 tx=0x7fc12c00fe10 comp rx=0 tx=0).stop 2023-07-14T19:18:34.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.156+0000 7fc13e5ef700 1 -- 172.21.15.104:0/3717892631 shutdown_connections 2023-07-14T19:18:34.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.156+0000 7fc13e5ef700 1 --2- 172.21.15.104:0/3717892631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1200057d0 0x7fc120007c50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:34.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.156+0000 7fc13e5ef700 1 -- 172.21.15.104:0/3717892631 >> 172.21.15.104:0/3717892631 conn(0x7fc1380fa290 msgr2=0x7fc1380fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:34.158 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.156+0000 7fc13e5ef700 1 -- 172.21.15.104:0/3717892631 shutdown_connections 2023-07-14T19:18:34.159 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.156+0000 7fc13e5ef700 1 -- 172.21.15.104:0/3717892631 wait complete. 2023-07-14T19:18:34.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.157+0000 7fc13e5ef700 1 Processor -- start 2023-07-14T19:18:34.160 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.157+0000 7fc13e5ef700 1 -- start start 2023-07-14T19:18:34.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.157+0000 7fc13e5ef700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1200057d0 0x7fc138192cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:34.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.157+0000 7fc13e5ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 0x7fc1381965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:34.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc13e5ef700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc138196b00 0x7fc138196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:34.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc13e5ef700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1381938a0 con 0x7fc1200057d0 2023-07-14T19:18:34.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc13e5ef700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13819f950 con 0x7fc138196b00 2023-07-14T19:18:34.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc13e5ef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13819fa90 con 0x7fc138193200 2023-07-14T19:18:34.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc1377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 0x7fc1381965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:34.161 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc1377fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 0x7fc1381965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56862/0 (socket says 172.21.15.104:56862) 2023-07-14T19:18:34.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc1377fe700 1 -- 172.21.15.104:0/1330848459 learned_addr learned my addr 172.21.15.104:0/1330848459 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:18:34.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc1377fe700 1 -- 172.21.15.104:0/1330848459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 msgr2=0x7fc1381965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T19:18:34.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc1377fe700 1 -- 172.21.15.104:0/1330848459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 msgr2=0x7fc1381965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T19:18:34.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc1377fe700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 0x7fc1381965c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T19:18:34.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc137fff700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1200057d0 0x7fc138192cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:34.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc1377fe700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 0x7fc1381965c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:18:34.162 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc13cb8c700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc138196b00 0x7fc138196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:34.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc13cb8c700 1 -- 172.21.15.104:0/1330848459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc138196b00 msgr2=0x7fc138196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-14T19:18:34.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc13cb8c700 1 -- 172.21.15.104:0/1330848459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc138196b00 msgr2=0x7fc138196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T19:18:34.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc13cb8c700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc138196b00 0x7fc138196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T19:18:34.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc13cb8c700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc138196b00 0x7fc138196f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:18:34.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc137fff700 1 -- 172.21.15.104:0/1330848459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 msgr2=0x7fc1381965c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:18:34.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc137fff700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 0x7fc1381965c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:34.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc137fff700 1 -- 172.21.15.104:0/1330848459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc138196b00 msgr2=0x7fc138196f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:18:34.163 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc137fff700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc138196b00 0x7fc138196f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:34.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc137fff700 1 -- 172.21.15.104:0/1330848459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c00f860 con 0x7fc1200057d0 2023-07-14T19:18:34.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.158+0000 7fc137fff700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1200057d0 0x7fc138192cc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc1380fe6a0 tx=0x7fc12c011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:34.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.159+0000 7fc1357fa700 1 -- 172.21.15.104:0/1330848459 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c0054d0 con 0x7fc1200057d0 2023-07-14T19:18:34.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.159+0000 7fc1357fa700 1 -- 172.21.15.104:0/1330848459 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc12c011eb0 con 0x7fc1200057d0 2023-07-14T19:18:34.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.159+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13819fcc0 con 0x7fc1200057d0 2023-07-14T19:18:34.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.159+0000 7fc1357fa700 1 -- 172.21.15.104:0/1330848459 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c01d8a0 con 0x7fc1200057d0 2023-07-14T19:18:34.164 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.159+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1381a01c0 con 0x7fc1200057d0 2023-07-14T19:18:34.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.160+0000 7fc1357fa700 1 -- 172.21.15.104:0/1330848459 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc12c024040 con 0x7fc1200057d0 2023-07-14T19:18:34.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.160+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc138061c70 con 0x7fc1200057d0 2023-07-14T19:18:34.165 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.160+0000 7fc1357fa700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc124038c30 0x7fc12403b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:34.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.160+0000 7fc1357fa700 1 -- 172.21.15.104:0/1330848459 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc12c04e460 con 0x7fc1200057d0 2023-07-14T19:18:34.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.160+0000 7fc1377fe700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc124038c30 0x7fc12403b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:34.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.161+0000 7fc1377fe700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc124038c30 0x7fc12403b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc128009180 tx=0x7fc128008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:34.166 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.165+0000 7fc1357fa700 1 -- 172.21.15.104:0/1330848459 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc12c0109d0 con 0x7fc1200057d0 2023-07-14T19:18:34.440 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.438+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc138105620 con 0x7fc1200057d0 2023-07-14T19:18:34.441 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.439+0000 7fc1357fa700 1 -- 172.21.15.104:0/1330848459 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1270 (secure 0 0 0) 0x7fc12c025e60 con 0x7fc1200057d0 2023-07-14T19:18:34.441 INFO:teuthology.orchestra.run.smithi104.stdout: 2023-07-14T19:18:34.441 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":3,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","modified":"2023-07-14T19:18:28.366044Z","created":"2023-07-14T19:16:45.602861Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T19:18:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc124038c30 msgr2=0x7fc12403b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc124038c30 0x7fc12403b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc128009180 tx=0x7fc128008040 comp rx=0 tx=0).stop 2023-07-14T19:18:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1200057d0 msgr2=0x7fc138192cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:34.443 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1200057d0 0x7fc138192cc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc1380fe6a0 tx=0x7fc12c011b90 comp rx=0 tx=0).stop 2023-07-14T19:18:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 shutdown_connections 2023-07-14T19:18:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc124038c30 0x7fc12403b0e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc138196b00 0x7fc138196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1200057d0 0x7fc138192cc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 --2- 172.21.15.104:0/1330848459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc138193200 0x7fc1381965c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 >> 172.21.15.104:0/1330848459 conn(0x7fc1380fa290 msgr2=0x7fc138100350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:34.444 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 shutdown_connections 2023-07-14T19:18:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:34.441+0000 7fc13e5ef700 1 -- 172.21.15.104:0/1330848459 wait complete. 2023-07-14T19:18:34.445 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 3 2023-07-14T19:18:34.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:34.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:34.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:34 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/1330848459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:18:34.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:34.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:34.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:34 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/1330848459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:18:34.844 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-14T19:18:34.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph config generate-minimal-conf 2023-07-14T19:18:35.419 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:18:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:18:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: mon.a calling monitor election 2023-07-14T19:18:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: mon.c calling monitor election 2023-07-14T19:18:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:18:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:35.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:35.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:35.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:18:35.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],c=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} 2023-07-14T19:18:35.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: fsmap 2023-07-14T19:18:35.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:35.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: mgrmap e13: x(active, since 52s) 2023-07-14T19:18:35.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T19:18:35.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T19:18:35.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T19:18:35.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: mon.b (rank 2) addr [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] is down (out of quorum) 2023-07-14T19:18:35.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:35.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:35.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:35.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:35.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:35.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:35.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:35 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/1330848459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:18:36.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: mon.b calling monitor election 2023-07-14T19:18:36.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: mon.b calling monitor election 2023-07-14T19:18:36.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: mon.c calling monitor election 2023-07-14T19:18:36.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: mon.a calling monitor election 2023-07-14T19:18:36.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T19:18:36.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],c=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} 2023-07-14T19:18:36.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: fsmap 2023-07-14T19:18:36.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:36.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: mgrmap e13: x(active, since 54s) 2023-07-14T19:18:36.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T19:18:36.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: Cluster is now healthy 2023-07-14T19:18:36.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: overall HEALTH_OK 2023-07-14T19:18:36.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:36.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:18:36.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:36.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: mon.b calling monitor election 2023-07-14T19:18:36.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: mon.b calling monitor election 2023-07-14T19:18:36.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: mon.c calling monitor election 2023-07-14T19:18:36.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: mon.a calling monitor election 2023-07-14T19:18:36.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T19:18:36.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],c=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} 2023-07-14T19:18:36.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: fsmap 2023-07-14T19:18:36.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:36.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: mgrmap e13: x(active, since 54s) 2023-07-14T19:18:36.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T19:18:36.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: Cluster is now healthy 2023-07-14T19:18:36.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: overall HEALTH_OK 2023-07-14T19:18:36.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:36.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:18:36.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:36.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: mon.b calling monitor election 2023-07-14T19:18:36.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: mon.b calling monitor election 2023-07-14T19:18:36.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: mon.c calling monitor election 2023-07-14T19:18:36.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: mon.a calling monitor election 2023-07-14T19:18:36.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T19:18:36.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0],c=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} 2023-07-14T19:18:36.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: fsmap 2023-07-14T19:18:36.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:18:36.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: mgrmap e13: x(active, since 54s) 2023-07-14T19:18:36.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T19:18:36.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: Cluster is now healthy 2023-07-14T19:18:36.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: overall HEALTH_OK 2023-07-14T19:18:36.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.514 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.514 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:36.514 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:36.514 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:18:36.514 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:36.725 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.723+0000 7f924eafd700 1 -- 172.21.15.53:0/627065265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92481008e0 msgr2=0x7f9248100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:36.725 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.723+0000 7f924eafd700 1 --2- 172.21.15.53:0/627065265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92481008e0 0x7f9248100cb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f923c009910 tx=0x7f923c009c20 comp rx=0 tx=0).stop 2023-07-14T19:18:36.725 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.723+0000 7f924eafd700 1 -- 172.21.15.53:0/627065265 shutdown_connections 2023-07-14T19:18:36.725 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.723+0000 7f924eafd700 1 --2- 172.21.15.53:0/627065265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92481008e0 0x7f9248100cb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:36.725 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.723+0000 7f924eafd700 1 -- 172.21.15.53:0/627065265 >> 172.21.15.53:0/627065265 conn(0x7f92480fa140 msgr2=0x7f92480fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:36.726 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.723+0000 7f924eafd700 1 -- 172.21.15.53:0/627065265 shutdown_connections 2023-07-14T19:18:36.726 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.723+0000 7f924eafd700 1 -- 172.21.15.53:0/627065265 wait complete. 2023-07-14T19:18:36.727 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924eafd700 1 Processor -- start 2023-07-14T19:18:36.728 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924eafd700 1 -- start start 2023-07-14T19:18:36.728 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924eafd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92481008e0 0x7f9248194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:36.728 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924eafd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9248195320 0x7f9248199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:36.728 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924eafd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9248199c80 0x7f924819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:36.728 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924eafd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92481959f0 con 0x7f9248195320 2023-07-14T19:18:36.728 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924eafd700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92481a19f0 con 0x7f9248199c80 2023-07-14T19:18:36.728 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924eafd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92481a1b30 con 0x7f92481008e0 2023-07-14T19:18:36.729 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f9247fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9248195320 0x7f9248199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:36.729 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f9247fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9248195320 0x7f9248199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49306/0 (socket says 172.21.15.53:49306) 2023-07-14T19:18:36.729 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f9247fff700 1 -- 172.21.15.53:0/3811288924 learned_addr learned my addr 172.21.15.53:0/3811288924 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:18:36.729 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924d09a700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9248199c80 0x7f924819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:36.729 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924c899700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92481008e0 0x7f9248194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:36.729 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f9247fff700 1 -- 172.21.15.53:0/3811288924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92481008e0 msgr2=0x7f9248194de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:36.729 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f9247fff700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92481008e0 0x7f9248194de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:36.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f9247fff700 1 -- 172.21.15.53:0/3811288924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9248199c80 msgr2=0x7f924819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:36.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f9247fff700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9248199c80 0x7f924819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:36.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f9247fff700 1 -- 172.21.15.53:0/3811288924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c0095f0 con 0x7f9248195320 2023-07-14T19:18:36.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.724+0000 7f924c899700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92481008e0 0x7f9248194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:18:36.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.725+0000 7f9247fff700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9248195320 0x7f9248199740 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f923400b7b0 tx=0x7f923400bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:36.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.725+0000 7f9245ffb700 1 -- 172.21.15.53:0/3811288924 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9234004230 con 0x7f9248195320 2023-07-14T19:18:36.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.725+0000 7f9245ffb700 1 -- 172.21.15.53:0/3811288924 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9234004390 con 0x7f9248195320 2023-07-14T19:18:36.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.725+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92481a1de0 con 0x7f9248195320 2023-07-14T19:18:36.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.725+0000 7f9245ffb700 1 -- 172.21.15.53:0/3811288924 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9234011ad0 con 0x7f9248195320 2023-07-14T19:18:36.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.725+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92481a2260 con 0x7f9248195320 2023-07-14T19:18:36.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.726+0000 7f9245ffb700 1 -- 172.21.15.53:0/3811288924 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9234011d10 con 0x7f9248195320 2023-07-14T19:18:36.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.726+0000 7f9245ffb700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9238038c30 0x7f923803b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:36.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.726+0000 7f9245ffb700 1 -- 172.21.15.53:0/3811288924 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f923404ad00 con 0x7f9248195320 2023-07-14T19:18:36.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.726+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9248107110 con 0x7f9248195320 2023-07-14T19:18:36.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.726+0000 7f924c899700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9238038c30 0x7f923803b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:36.732 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.727+0000 7f924c899700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9238038c30 0x7f923803b0e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f923c000c00 tx=0x7f923c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:36.732 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.731+0000 7f9245ffb700 1 -- 172.21.15.53:0/3811288924 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f923401a400 con 0x7f9248195320 2023-07-14T19:18:36.943 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.941+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f924802c8e0 con 0x7f9248195320 2023-07-14T19:18:36.943 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.942+0000 7f9245ffb700 1 -- 172.21.15.53:0/3811288924 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7f9234048030 con 0x7f9248195320 2023-07-14T19:18:36.944 INFO:teuthology.orchestra.run.smithi053.stdout:# minimal ceph.conf for c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:18:36.944 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2023-07-14T19:18:36.944 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:18:36.944 INFO:teuthology.orchestra.run.smithi053.stdout: mon_host = [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2023-07-14T19:18:36.945 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9238038c30 msgr2=0x7f923803b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:36.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9238038c30 0x7f923803b0e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f923c000c00 tx=0x7f923c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:18:36.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9248195320 msgr2=0x7f9248199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:36.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9248195320 0x7f9248199740 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f923400b7b0 tx=0x7f923400bb70 comp rx=0 tx=0).stop 2023-07-14T19:18:36.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 shutdown_connections 2023-07-14T19:18:36.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9238038c30 0x7f923803b0e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:36.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9248199c80 0x7f924819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:36.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92481008e0 0x7f9248194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:36.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 --2- 172.21.15.53:0/3811288924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9248195320 0x7f9248199740 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:36.947 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 >> 172.21.15.53:0/3811288924 conn(0x7f92480fa140 msgr2=0x7f9248104250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:36.947 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 shutdown_connections 2023-07-14T19:18:36.947 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:36.944+0000 7f924eafd700 1 -- 172.21.15.53:0/3811288924 wait complete. 2023-07-14T19:18:37.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:37 smithi070 ceph-mon[98015]: Updating smithi053:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:37.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:37 smithi070 ceph-mon[98015]: Updating smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:37.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:37 smithi070 ceph-mon[98015]: Updating smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:37.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:37 smithi070 ceph-mon[98015]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T19:18:37.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:37 smithi070 ceph-mon[98015]: Reconfiguring daemon mon.a on smithi053 2023-07-14T19:18:37.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:37 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:37.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:37 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:37.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:37 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3811288924' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:37.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:37 smithi104 ceph-mon[98435]: Updating smithi053:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:37.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:37 smithi104 ceph-mon[98435]: Updating smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:37.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:37 smithi104 ceph-mon[98435]: Updating smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:37.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:37 smithi104 ceph-mon[98435]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T19:18:37.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:37 smithi104 ceph-mon[98435]: Reconfiguring daemon mon.a on smithi053 2023-07-14T19:18:37.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:37 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:37.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:37 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:37.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:37 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3811288924' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:37.557 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:37 smithi053 ceph-mon[96565]: Updating smithi053:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:37.557 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:37 smithi053 ceph-mon[96565]: Updating smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:37.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:37 smithi053 ceph-mon[96565]: Updating smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:18:37.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:37 smithi053 ceph-mon[96565]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T19:18:37.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:37 smithi053 ceph-mon[96565]: Reconfiguring daemon mon.a on smithi053 2023-07-14T19:18:37.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:37 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:37.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:37 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:18:37.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:37 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3811288924' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:37.575 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-14T19:18:37.575 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:18:37.575 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T19:18:37.606 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:18:37.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:37.673 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:18:37.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T19:18:37.706 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:18:37.707 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:37.773 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:18:37.773 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T19:18:37.810 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:18:37.810 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:18:37.881 INFO:tasks.cephadm:Adding mgr.x on smithi053 2023-07-14T19:18:37.881 INFO:tasks.cephadm:Adding mgr.y on smithi070 2023-07-14T19:18:37.881 INFO:tasks.cephadm:Adding mgr.z on smithi104 2023-07-14T19:18:37.881 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch apply mgr '3;smithi053=x;smithi070=y;smithi104=z' 2023-07-14T19:18:38.473 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:18:38.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:38 smithi070 ceph-mon[98015]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:38.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:38 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:38.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:38 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:38.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:38 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:38.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:38 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:18:38.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:38 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:38.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:38 smithi104 ceph-mon[98435]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:38.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:38 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:38.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:38 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:38.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:38 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:38.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:38 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:18:38.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:38 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:38 smithi053 ceph-mon[96565]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:38 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:38 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:38 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:38 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:18:38.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:38 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:39.438 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:39 smithi104 ceph-mon[98435]: Reconfiguring mon.b (monmap changed)... 2023-07-14T19:18:39.439 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:39 smithi104 ceph-mon[98435]: Reconfiguring daemon mon.b on smithi070 2023-07-14T19:18:39.439 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:39.439 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:39.439 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:39.439 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:18:39.440 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:39.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:39 smithi070 ceph-mon[98015]: Reconfiguring mon.b (monmap changed)... 2023-07-14T19:18:39.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:39 smithi070 ceph-mon[98015]: Reconfiguring daemon mon.b on smithi070 2023-07-14T19:18:39.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:39.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:39.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:39.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:18:39.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:39 smithi053 ceph-mon[96565]: Reconfiguring mon.b (monmap changed)... 2023-07-14T19:18:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:39 smithi053 ceph-mon[96565]: Reconfiguring daemon mon.b on smithi070 2023-07-14T19:18:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:18:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:18:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:40.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.085+0000 7f4e1c589700 1 -- 172.21.15.104:0/4161714093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 msgr2=0x7f4e141001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:40.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.085+0000 7f4e1c589700 1 --2- 172.21.15.104:0/4161714093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 0x7f4e141001d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009910 tx=0x7f4e08009c20 comp rx=0 tx=0).stop 2023-07-14T19:18:40.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.085+0000 7f4e1c589700 1 -- 172.21.15.104:0/4161714093 shutdown_connections 2023-07-14T19:18:40.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.085+0000 7f4e1c589700 1 --2- 172.21.15.104:0/4161714093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 0x7f4e141001d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:40.087 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.085+0000 7f4e1c589700 1 -- 172.21.15.104:0/4161714093 >> 172.21.15.104:0/4161714093 conn(0x7f4e140f9450 msgr2=0x7f4e140fb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:40.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.085+0000 7f4e1c589700 1 -- 172.21.15.104:0/4161714093 shutdown_connections 2023-07-14T19:18:40.088 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.085+0000 7f4e1c589700 1 -- 172.21.15.104:0/4161714093 wait complete. 2023-07-14T19:18:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1c589700 1 Processor -- start 2023-07-14T19:18:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1c589700 1 -- start start 2023-07-14T19:18:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1c589700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 0x7f4e140fb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1c589700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e140f9a50 0x7f4e140f9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1c589700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e140fa400 0x7f4e1419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1c589700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1419f180 con 0x7f4e140ffdc0 2023-07-14T19:18:40.090 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1c589700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1419f2c0 con 0x7f4e140f9a50 2023-07-14T19:18:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1c589700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1419f400 con 0x7f4e140fa400 2023-07-14T19:18:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1a325700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 0x7f4e140fb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1a325700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 0x7f4e140fb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.104:49164/0 (socket says 172.21.15.104:49164) 2023-07-14T19:18:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1a325700 1 -- 172.21.15.104:0/2271203639 learned_addr learned my addr 172.21.15.104:0/2271203639 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:18:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e19b24700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e140f9a50 0x7f4e140f9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1ab26700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e140fa400 0x7f4e1419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:40.091 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1a325700 1 -- 172.21.15.104:0/2271203639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e140fa400 msgr2=0x7f4e1419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1a325700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e140fa400 0x7f4e1419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1a325700 1 -- 172.21.15.104:0/2271203639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e140f9a50 msgr2=0x7f4e140f9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1a325700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e140f9a50 0x7f4e140f9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1a325700 1 -- 172.21.15.104:0/2271203639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e080095f0 con 0x7f4e140ffdc0 2023-07-14T19:18:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e19b24700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e140f9a50 0x7f4e140f9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:18:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.086+0000 7f4e1ab26700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e140fa400 0x7f4e1419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:18:40.092 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.087+0000 7f4e1a325700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 0x7f4e140fb400 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4e08004cb0 tx=0x7f4e08004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.087+0000 7f4e077fe700 1 -- 172.21.15.104:0/2271203639 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0801d070 con 0x7f4e140ffdc0 2023-07-14T19:18:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.087+0000 7f4e077fe700 1 -- 172.21.15.104:0/2271203639 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e0800be20 con 0x7f4e140ffdc0 2023-07-14T19:18:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.087+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1419f680 con 0x7f4e140ffdc0 2023-07-14T19:18:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.087+0000 7f4e077fe700 1 -- 172.21.15.104:0/2271203639 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0800f460 con 0x7f4e140ffdc0 2023-07-14T19:18:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.087+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1419fb20 con 0x7f4e140ffdc0 2023-07-14T19:18:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.088+0000 7f4e077fe700 1 -- 172.21.15.104:0/2271203639 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4e0800b960 con 0x7f4e140ffdc0 2023-07-14T19:18:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.088+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1418e380 con 0x7f4e140ffdc0 2023-07-14T19:18:40.093 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.088+0000 7f4e077fe700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4e00038c30 0x7f4e0003b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:40.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.088+0000 7f4e077fe700 1 -- 172.21.15.104:0/2271203639 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4e0804c7a0 con 0x7f4e140ffdc0 2023-07-14T19:18:40.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.088+0000 7f4e19b24700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4e00038c30 0x7f4e0003b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:40.094 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.089+0000 7f4e19b24700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4e00038c30 0x7f4e0003b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009910 tx=0x7f4e10008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:40.095 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.093+0000 7f4e077fe700 1 -- 172.21.15.104:0/2271203639 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4e0800ea30 con 0x7f4e140ffdc0 2023-07-14T19:18:40.296 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.295+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=x;smithi070=y;smithi104=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e14060b10 con 0x7f4e00038c30 2023-07-14T19:18:40.303 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.302+0000 7f4e077fe700 1 -- 172.21.15.104:0/2271203639 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4e14060b10 con 0x7f4e00038c30 2023-07-14T19:18:40.304 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mgr update... 2023-07-14T19:18:40.305 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4e00038c30 msgr2=0x7f4e0003b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:40.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4e00038c30 0x7f4e0003b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4e10009910 tx=0x7f4e10008040 comp rx=0 tx=0).stop 2023-07-14T19:18:40.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 msgr2=0x7f4e140fb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:40.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 0x7f4e140fb400 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4e08004cb0 tx=0x7f4e08004d90 comp rx=0 tx=0).stop 2023-07-14T19:18:40.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 shutdown_connections 2023-07-14T19:18:40.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4e00038c30 0x7f4e0003b0e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:40.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e140fa400 0x7f4e1419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:40.306 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e140ffdc0 0x7f4e140fb400 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:40.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 --2- 172.21.15.104:0/2271203639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e140f9a50 0x7f4e140f9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:40.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 >> 172.21.15.104:0/2271203639 conn(0x7f4e140f9450 msgr2=0x7f4e140fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:40.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 shutdown_connections 2023-07-14T19:18:40.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:18:40.304+0000 7f4e1c589700 1 -- 172.21.15.104:0/2271203639 wait complete. 2023-07-14T19:18:40.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:40 smithi070 ceph-mon[98015]: Reconfiguring mon.c (monmap changed)... 2023-07-14T19:18:40.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:40 smithi070 ceph-mon[98015]: Reconfiguring daemon mon.c on smithi104 2023-07-14T19:18:40.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:40 smithi070 ceph-mon[98015]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:40.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:40 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:40.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:40 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:40.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:40 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:40.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:40 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:40.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:40 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:40.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:40 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:40.480 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:40 smithi104 ceph-mon[98435]: Reconfiguring mon.c (monmap changed)... 2023-07-14T19:18:40.481 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:40 smithi104 ceph-mon[98435]: Reconfiguring daemon mon.c on smithi104 2023-07-14T19:18:40.481 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:40 smithi104 ceph-mon[98435]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:40.481 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:40 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:40.481 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:40 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:40.482 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:40 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:40.482 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:40 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:40.482 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:40 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:40.482 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:40 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:40.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:40 smithi053 ceph-mon[96565]: Reconfiguring mon.c (monmap changed)... 2023-07-14T19:18:40.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:40 smithi053 ceph-mon[96565]: Reconfiguring daemon mon.c on smithi104 2023-07-14T19:18:40.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:40 smithi053 ceph-mon[96565]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:40.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:40 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:40.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:40 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:40.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:40 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:40.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:40 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:40.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:40 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:40.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:40 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:40.773 DEBUG:teuthology.orchestra.run.smithi070:mgr.y> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.y.service 2023-07-14T19:18:40.775 DEBUG:teuthology.orchestra.run.smithi104:mgr.z> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.z.service 2023-07-14T19:18:40.777 INFO:tasks.cephadm:Deploying OSDs... 2023-07-14T19:18:40.778 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:18:40.778 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T19:18:40.795 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T19:18:40.796 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2023-07-14T19:18:40.807 INFO:journalctl@ceph.mgr.z.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:18:40.810 INFO:journalctl@ceph.mgr.y.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:18:40.852 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T19:18:40.853 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:40.853 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 6h/6d Inode: 380050 Links: 1 2023-07-14T19:18:40.853 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:40.853 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:40.853 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-07-14 19:17:52.136993682 +0000 2023-07-14T19:18:40.853 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-07-14 19:04:59.631366472 +0000 2023-07-14T19:18:40.854 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-07-14 19:04:59.631366472 +0000 2023-07-14T19:18:40.854 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2023-07-14T19:18:40.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T19:18:40.920 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-07-14T19:18:40.921 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-07-14T19:18:40.921 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.2504e-05 s, 5.5 MB/s 2023-07-14T19:18:40.922 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T19:18:40.983 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2023-07-14T19:18:41.041 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T19:18:41.041 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:41.042 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 6h/6d Inode: 380211 Links: 1 2023-07-14T19:18:41.042 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:41.042 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:41.042 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-07-14 19:17:52.136993682 +0000 2023-07-14T19:18:41.042 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-07-14 19:05:00.163355400 +0000 2023-07-14T19:18:41.042 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-07-14 19:05:00.163355400 +0000 2023-07-14T19:18:41.042 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2023-07-14T19:18:41.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T19:18:41.108 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-07-14T19:18:41.108 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-07-14T19:18:41.108 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000104573 s, 4.9 MB/s 2023-07-14T19:18:41.109 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T19:18:41.167 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2023-07-14T19:18:41.225 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T19:18:41.225 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:41.225 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 6h/6d Inode: 382168 Links: 1 2023-07-14T19:18:41.225 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:41.226 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:41.226 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-07-14 19:17:52.136993682 +0000 2023-07-14T19:18:41.226 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-07-14 19:05:00.676344723 +0000 2023-07-14T19:18:41.226 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-07-14 19:05:00.676344723 +0000 2023-07-14T19:18:41.226 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2023-07-14T19:18:41.226 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T19:18:41.293 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-07-14T19:18:41.293 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-07-14T19:18:41.294 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000110796 s, 4.6 MB/s 2023-07-14T19:18:41.294 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T19:18:41.353 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2023-07-14T19:18:41.412 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T19:18:41.412 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:41.412 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 6h/6d Inode: 382240 Links: 1 2023-07-14T19:18:41.412 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:41.412 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:41.412 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-07-14 19:17:52.136993682 +0000 2023-07-14T19:18:41.413 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-07-14 19:05:01.196333901 +0000 2023-07-14T19:18:41.413 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-07-14 19:05:01.196333901 +0000 2023-07-14T19:18:41.413 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: - 2023-07-14T19:18:41.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T19:18:41.468 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:41 smithi053 ceph-mon[96565]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=x;smithi070=y;smithi104=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:41.468 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:41 smithi053 ceph-mon[96565]: Saving service mgr spec with placement smithi053=x;smithi070=y;smithi104=z;count:3 2023-07-14T19:18:41.468 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:41.468 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:41 smithi053 ceph-mon[96565]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T19:18:41.469 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:18:41.469 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:18:41.469 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:41.469 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:41 smithi053 ceph-mon[96565]: Reconfiguring daemon mgr.x on smithi053 2023-07-14T19:18:41.469 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:41 smithi053 ceph-mon[96565]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:41.478 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-07-14T19:18:41.479 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-07-14T19:18:41.479 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000110328 s, 4.6 MB/s 2023-07-14T19:18:41.480 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T19:18:41.538 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2023-07-14T19:18:41.538 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T19:18:41.559 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T19:18:41.559 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2023-07-14T19:18:41.619 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T19:18:41.620 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:41.620 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 6h/6d Inode: 373180 Links: 1 2023-07-14T19:18:41.620 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:41.620 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:41.621 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2023-07-14 19:18:27.818869088 +0000 2023-07-14T19:18:41.621 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2023-07-14 19:04:07.233718434 +0000 2023-07-14T19:18:41.621 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2023-07-14 19:04:07.233718434 +0000 2023-07-14T19:18:41.621 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2023-07-14T19:18:41.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T19:18:41.686 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:41 smithi070 ceph-mon[98015]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=x;smithi070=y;smithi104=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:41.687 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:41 smithi070 ceph-mon[98015]: Saving service mgr spec with placement smithi053=x;smithi070=y;smithi104=z;count:3 2023-07-14T19:18:41.687 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:41.687 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:41 smithi070 ceph-mon[98015]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T19:18:41.687 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:18:41.687 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:18:41.687 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:41.688 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:41 smithi070 ceph-mon[98015]: Reconfiguring daemon mgr.x on smithi053 2023-07-14T19:18:41.688 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:41 smithi070 ceph-mon[98015]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:41.689 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2023-07-14T19:18:41.689 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2023-07-14T19:18:41.689 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.00012104 s, 4.2 MB/s 2023-07-14T19:18:41.690 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T19:18:41.750 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2023-07-14T19:18:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:41 smithi104 ceph-mon[98435]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=x;smithi070=y;smithi104=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:41 smithi104 ceph-mon[98435]: Saving service mgr spec with placement smithi053=x;smithi070=y;smithi104=z;count:3 2023-07-14T19:18:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:41 smithi104 ceph-mon[98435]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T19:18:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:18:41.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:18:41.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:41.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:41 smithi104 ceph-mon[98435]: Reconfiguring daemon mgr.x on smithi053 2023-07-14T19:18:41.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:41 smithi104 ceph-mon[98435]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T19:18:41.809 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:41.809 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 6h/6d Inode: 373228 Links: 1 2023-07-14T19:18:41.810 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:41.810 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:41.810 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2023-07-14 19:18:27.819869058 +0000 2023-07-14T19:18:41.810 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2023-07-14 19:04:07.749702743 +0000 2023-07-14T19:18:41.810 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2023-07-14 19:04:07.749702743 +0000 2023-07-14T19:18:41.810 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2023-07-14T19:18:41.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T19:18:41.877 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2023-07-14T19:18:41.877 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2023-07-14T19:18:41.877 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.00011865 s, 4.3 MB/s 2023-07-14T19:18:41.878 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T19:18:41.937 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2023-07-14T19:18:41.996 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T19:18:41.996 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:41.996 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 6h/6d Inode: 380439 Links: 1 2023-07-14T19:18:41.997 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:41.997 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:41.997 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2023-07-14 19:18:27.819869058 +0000 2023-07-14T19:18:41.997 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2023-07-14 19:04:08.259687235 +0000 2023-07-14T19:18:41.997 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2023-07-14 19:04:08.259687235 +0000 2023-07-14T19:18:41.997 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2023-07-14T19:18:41.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T19:18:42.063 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2023-07-14T19:18:42.063 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2023-07-14T19:18:42.064 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000120176 s, 4.3 MB/s 2023-07-14T19:18:42.064 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T19:18:42.121 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2023-07-14T19:18:42.178 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T19:18:42.179 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:42.179 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 6h/6d Inode: 376457 Links: 1 2023-07-14T19:18:42.179 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:42.179 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:42.179 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2023-07-14 19:18:27.819869058 +0000 2023-07-14T19:18:42.180 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2023-07-14 19:04:08.754672184 +0000 2023-07-14T19:18:42.180 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2023-07-14 19:04:08.754672184 +0000 2023-07-14T19:18:42.180 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2023-07-14T19:18:42.180 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T19:18:42.244 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2023-07-14T19:18:42.244 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2023-07-14T19:18:42.245 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000119831 s, 4.3 MB/s 2023-07-14T19:18:42.246 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T19:18:42.303 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2023-07-14T19:18:42.303 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T19:18:42.321 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T19:18:42.321 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2023-07-14T19:18:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T19:18:42.378 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:42.378 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 378412 Links: 1 2023-07-14T19:18:42.379 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:42.379 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:42.379 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2023-07-14 19:18:19.559684406 +0000 2023-07-14T19:18:42.379 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2023-07-14 19:04:07.231177601 +0000 2023-07-14T19:18:42.379 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2023-07-14 19:04:07.231177601 +0000 2023-07-14T19:18:42.379 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2023-07-14T19:18:42.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T19:18:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2023-07-14T19:18:42.445 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2023-07-14T19:18:42.446 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000222811 s, 2.3 MB/s 2023-07-14T19:18:42.446 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T19:18:42.503 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2023-07-14T19:18:42.560 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T19:18:42.561 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:42.561 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 369572 Links: 1 2023-07-14T19:18:42.561 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:42.561 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:42.561 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2023-07-14 19:18:19.559684406 +0000 2023-07-14T19:18:42.561 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2023-07-14 19:04:07.795170940 +0000 2023-07-14T19:18:42.562 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2023-07-14 19:04:07.795170940 +0000 2023-07-14T19:18:42.562 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2023-07-14T19:18:42.562 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T19:18:42.625 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2023-07-14T19:18:42.626 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2023-07-14T19:18:42.626 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000199373 s, 2.6 MB/s 2023-07-14T19:18:42.628 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T19:18:42.685 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2023-07-14T19:18:42.743 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T19:18:42.743 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:42.744 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 375509 Links: 1 2023-07-14T19:18:42.744 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:42.744 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:42.745 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2023-07-14 19:18:19.559684406 +0000 2023-07-14T19:18:42.745 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2023-07-14 19:04:08.300164976 +0000 2023-07-14T19:18:42.745 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2023-07-14 19:04:08.300164976 +0000 2023-07-14T19:18:42.745 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2023-07-14T19:18:42.746 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T19:18:42.806 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:42.807 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:42.807 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:42.807 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:42.807 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:42.807 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:42.808 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:18:42.808 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:18:42.808 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:18:42.808 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:42.808 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:42 smithi104 ceph-mon[98435]: Deploying daemon mgr.z on smithi104 2023-07-14T19:18:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2023-07-14T19:18:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2023-07-14T19:18:42.809 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000215954 s, 2.4 MB/s 2023-07-14T19:18:42.810 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T19:18:42.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:42.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:42.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:42.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:42.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:42.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:42.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:18:42.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:18:42.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:18:42.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:42.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:42 smithi053 ceph-mon[96565]: Deploying daemon mgr.z on smithi104 2023-07-14T19:18:42.869 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2023-07-14T19:18:42.937 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T19:18:42.938 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:18:42.938 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 382026 Links: 1 2023-07-14T19:18:42.938 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:18:42.938 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:18:42.938 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2023-07-14 19:18:19.559684406 +0000 2023-07-14T19:18:42.939 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2023-07-14 19:04:08.799159083 +0000 2023-07-14T19:18:42.939 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2023-07-14 19:04:08.799159083 +0000 2023-07-14T19:18:42.939 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2023-07-14T19:18:42.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T19:18:42.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:42.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:42.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:42.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:42.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:42.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:42.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:18:42.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:18:42.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:18:42.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:42.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:42 smithi070 ceph-mon[98015]: Deploying daemon mgr.z on smithi104 2023-07-14T19:18:43.007 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2023-07-14T19:18:43.008 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2023-07-14T19:18:43.008 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000177139 s, 2.9 MB/s 2023-07-14T19:18:43.009 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T19:18:43.072 INFO:tasks.cephadm:Deploying osd.0 on smithi053 with /dev/vg_nvme/lv_4... 2023-07-14T19:18:43.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T19:18:43.136 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:43 smithi104 systemd[1]: Starting Ceph mgr.z for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:18:43.642 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:18:43.724 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:43 smithi104 ceph-mon[98435]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:43.724 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:43 smithi104 podman[100577]: 2023-07-14T19:18:43.724 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:43 smithi104 bash[100577]: 9c478561191767e9d68a091715e477642d4c09303696f274f7abbe85bc4726f1 2023-07-14T19:18:43.724 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:43 smithi104 ceph-mgr[100617]: -- 172.21.15.104:0/804898199 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558c4e9f71e0 con 0x558c4e9cd000 2023-07-14T19:18:43.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:43 smithi053 ceph-mon[96565]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:43.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:43 smithi070 ceph-mon[98015]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:44.011 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:43 smithi104 systemd[1]: Started Ceph mgr.z for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:18:44.011 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:43 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:43.798+0000 7fc025702200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:18:44.011 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:43 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:43.877+0000 7fc025702200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:18:44.011 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:43 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:43.964+0000 7fc025702200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:18:44.511 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:44.056+0000 7fc025702200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:18:44.511 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:44.224+0000 7fc025702200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:18:44.788 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:44.527+0000 7fc025702200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:18:44.788 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:44.615+0000 7fc025702200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:18:44.789 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:44.742+0000 7fc025702200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:18:45.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:45.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:45.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:45.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:18:45.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:18:45.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:18:45.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:45.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:44 smithi053 ceph-mon[96565]: Deploying daemon mgr.y on smithi070 2023-07-14T19:18:45.074 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:45.074 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:45.074 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:45.075 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:18:45.075 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:18:45.075 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:18:45.075 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:45.075 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:44 smithi070 ceph-mon[98015]: Deploying daemon mgr.y on smithi070 2023-07-14T19:18:45.155 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T19:18:45.155 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T19:18:45.155 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2023-07-14T19:18:45.156 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2023-07-14T19:18:45.156 INFO:teuthology.orchestra.run.smithi053.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00816614 s, 1.3 GB/s 2023-07-14T19:18:45.156 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2023-07-14T19:18:45.252 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:44.836+0000 7fc025702200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:18:45.252 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:45.253 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:45.253 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:45.253 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:18:45.253 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:18:45.253 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:18:45.253 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:45.253 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:44 smithi104 ceph-mon[98435]: Deploying daemon mgr.y on smithi070 2023-07-14T19:18:45.502 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi053:vg_nvme/lv_4 2023-07-14T19:18:45.510 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:45 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:45.250+0000 7fc025702200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:18:45.639 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:45 smithi070 systemd[1]: Starting Ceph mgr.y for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:18:45.899 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:45 smithi070 ceph-mon[98015]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:45.899 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:45 smithi070 podman[99480]: 2023-07-14T19:18:46.063 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:18:46.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:45 smithi053 ceph-mon[96565]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:46.195 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:45 smithi070 ceph-mgr[99520]: -- 172.21.15.70:0/3331542807 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559b99b811e0 con 0x559b99b56c00 2023-07-14T19:18:46.195 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:45 smithi070 bash[99480]: 2664713cf13ba013d8a990a21a9be3946302a3afc699d348a1f83676fb21048e 2023-07-14T19:18:46.195 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:45 smithi070 systemd[1]: Started Ceph mgr.y for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:18:46.195 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:46 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:46.019+0000 7ff46f1c8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:18:46.196 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:46 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:46.098+0000 7ff46f1c8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:18:46.196 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:46 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:46.193+0000 7ff46f1c8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:18:46.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:45 smithi104 ceph-mon[98435]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:46.261 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:45 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:45.821+0000 7fc025702200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:18:46.261 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:45 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:45.988+0000 7fc025702200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:18:46.475 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:46 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:46.295+0000 7ff46f1c8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:18:46.728 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:46 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:46.473+0000 7ff46f1c8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:18:47.045 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:46 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:46.772+0000 7fc025702200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:18:47.046 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:46 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:46.869+0000 7fc025702200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:18:47.046 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.046 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.046 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.046 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.047 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:47.047 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.123 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:46 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:46.792+0000 7ff46f1c8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:18:47.124 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:46 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:46.887+0000 7ff46f1c8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:18:47.124 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:47 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:47.025+0000 7ff46f1c8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:18:47.124 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:47 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:47 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:47 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:47 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:47 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:47.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:47 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.273+0000 7fcd4c504700 1 -- 172.21.15.53:0/730889532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44103060 msgr2=0x7fcd44105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.273+0000 7fcd4c504700 1 --2- 172.21.15.53:0/730889532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44103060 0x7fcd44105440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcd34009990 tx=0x7fcd34009ca0 comp rx=0 tx=0).stop 2023-07-14T19:18:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.273+0000 7fcd4c504700 1 -- 172.21.15.53:0/730889532 shutdown_connections 2023-07-14T19:18:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.273+0000 7fcd4c504700 1 --2- 172.21.15.53:0/730889532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd44105980 0x7fcd44107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.273+0000 7fcd4c504700 1 --2- 172.21.15.53:0/730889532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd44100740 0x7fcd44102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.273+0000 7fcd4c504700 1 --2- 172.21.15.53:0/730889532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44103060 0x7fcd44105440 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.273+0000 7fcd4c504700 1 -- 172.21.15.53:0/730889532 >> 172.21.15.53:0/730889532 conn(0x7fcd440fa1e0 msgr2=0x7fcd440fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:18:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.273+0000 7fcd4c504700 1 -- 172.21.15.53:0/730889532 shutdown_connections 2023-07-14T19:18:47.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.273+0000 7fcd4c504700 1 -- 172.21.15.53:0/730889532 wait complete. 2023-07-14T19:18:47.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4c504700 1 Processor -- start 2023-07-14T19:18:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4c504700 1 -- start start 2023-07-14T19:18:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4c504700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44100740 0x7fcd441000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4c504700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd44103060 0x7fcd440fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4c504700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd44105980 0x7fcd440fec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4c504700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440ff3b0 con 0x7fcd44100740 2023-07-14T19:18:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4c504700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440ff520 con 0x7fcd44103060 2023-07-14T19:18:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4c504700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4419fa70 con 0x7fcd44105980 2023-07-14T19:18:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4a2a0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44100740 0x7fcd441000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4a2a0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44100740 0x7fcd441000a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49316/0 (socket says 172.21.15.53:49316) 2023-07-14T19:18:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4a2a0700 1 -- 172.21.15.53:0/3730957652 learned_addr learned my addr 172.21.15.53:0/3730957652 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:18:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd49a9f700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd44103060 0x7fcd440fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4aaa1700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd44105980 0x7fcd440fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4a2a0700 1 -- 172.21.15.53:0/3730957652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd44105980 msgr2=0x7fcd440fec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4a2a0700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd44105980 0x7fcd440fec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4a2a0700 1 -- 172.21.15.53:0/3730957652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd44103060 msgr2=0x7fcd440fe6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:18:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4a2a0700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd44103060 0x7fcd440fe6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:18:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.274+0000 7fcd4a2a0700 1 -- 172.21.15.53:0/3730957652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd340095f0 con 0x7fcd44100740 2023-07-14T19:18:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.275+0000 7fcd4a2a0700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44100740 0x7fcd441000a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000d980 tx=0x7fcd4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.275+0000 7fcd3b7fe700 1 -- 172.21.15.53:0/3730957652 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40004220 con 0x7fcd44100740 2023-07-14T19:18:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.275+0000 7fcd3b7fe700 1 -- 172.21.15.53:0/3730957652 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd40007640 con 0x7fcd44100740 2023-07-14T19:18:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.275+0000 7fcd3b7fe700 1 -- 172.21.15.53:0/3730957652 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40005690 con 0x7fcd44100740 2023-07-14T19:18:47.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.275+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4419fd50 con 0x7fcd44100740 2023-07-14T19:18:47.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.275+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd441a01a0 con 0x7fcd44100740 2023-07-14T19:18:47.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.276+0000 7fcd3b7fe700 1 -- 172.21.15.53:0/3730957652 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fcd40004750 con 0x7fcd44100740 2023-07-14T19:18:47.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.276+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4410b4f0 con 0x7fcd44100740 2023-07-14T19:18:47.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.276+0000 7fcd3b7fe700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcd300389e0 0x7fcd3003ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:18:47.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.276+0000 7fcd3b7fe700 1 -- 172.21.15.53:0/3730957652 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcd4004b0a0 con 0x7fcd44100740 2023-07-14T19:18:47.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.276+0000 7fcd49a9f700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcd300389e0 0x7fcd3003ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:18:47.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.276+0000 7fcd49a9f700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcd300389e0 0x7fcd3003ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400b2e0 tx=0x7fcd34005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:18:47.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.281+0000 7fcd3b7fe700 1 -- 172.21.15.53:0/3730957652 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcd4000ed80 con 0x7fcd44100740 2023-07-14T19:18:47.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:47 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:47 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:47 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:47 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:47 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:47.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:47 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:47.353 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:47.044+0000 7fc025702200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:18:47.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:47.488+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd44102300 con 0x7fcd300389e0 2023-07-14T19:18:47.752 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:47.351+0000 7fc025702200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:18:47.752 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:47.597+0000 7fc025702200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:18:47.978 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:47 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:47.121+0000 7ff46f1c8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:18:47.978 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:47 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:47.573+0000 7ff46f1c8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:18:48.010 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:47 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:47.750+0000 7fc025702200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:18:48.478 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:48.159+0000 7ff46f1c8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:18:48.478 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:48.327+0000 7ff46f1c8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:18:48.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:48.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:18:48.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:18:48.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:48.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:48.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:48.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:48.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:48.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:48.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.487 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:48.488 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:18:48.488 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:18:48.488 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:48.488 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.489 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.489 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.489 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.489 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:48.490 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:48.490 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.490 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:48.490 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:48.491 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:48.491 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.511 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:48.013+0000 7fc025702200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:18:48.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:48.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:18:48.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:18:48.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:48.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:48.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:18:48.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:48.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:18:48.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:18:49.010 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:48.712+0000 7fc025702200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:18:49.011 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:48 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:48.841+0000 7fc025702200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:18:49.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:49 smithi053 ceph-mon[96565]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:49.350 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:49 smithi070 ceph-mon[98015]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:49.351 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:49 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:49.087+0000 7ff46f1c8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:18:49.351 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:49 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:49.178+0000 7ff46f1c8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:18:49.351 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:49 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:49.348+0000 7ff46f1c8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:18:49.362 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:49 smithi104 ceph-mon[98435]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:18:49.363 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:49.031+0000 7fc025702200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:18:49.363 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:49.145+0000 7fc025702200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:18:49.761 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:49.361+0000 7fc025702200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:18:49.978 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:49 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:49.622+0000 7ff46f1c8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:18:49.978 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:49 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:49.867+0000 7ff46f1c8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:18:50.276 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:50 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:50.015+0000 7ff46f1c8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:18:50.277 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:50 smithi070 ceph-mon[98015]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:50.277 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:50 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/169678783' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c120ad-b975-4c88-b14e-4f72a0d61a3b"}]: dispatch 2023-07-14T19:18:50.277 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:50 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/169678783' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14c120ad-b975-4c88-b14e-4f72a0d61a3b"}]': finished 2023-07-14T19:18:50.277 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:50 smithi070 ceph-mon[98015]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T19:18:50.277 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:18:50.511 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:18:50 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z[100593]: 2023-07-14T19:18:50.206+0000 7fc025702200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:18:50.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:50 smithi104 ceph-mon[98435]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:50.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:50 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/169678783' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c120ad-b975-4c88-b14e-4f72a0d61a3b"}]: dispatch 2023-07-14T19:18:50.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:50 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/169678783' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14c120ad-b975-4c88-b14e-4f72a0d61a3b"}]': finished 2023-07-14T19:18:50.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:50 smithi104 ceph-mon[98435]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T19:18:50.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:18:50.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:50 smithi053 ceph-mon[96565]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:50.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:50 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/169678783' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14c120ad-b975-4c88-b14e-4f72a0d61a3b"}]: dispatch 2023-07-14T19:18:50.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:50 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/169678783' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14c120ad-b975-4c88-b14e-4f72a0d61a3b"}]': finished 2023-07-14T19:18:50.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:50 smithi053 ceph-mon[96565]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T19:18:50.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:18:50.728 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:50 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:50.274+0000 7ff46f1c8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:18:51.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:51.225+0000 7fcd3b7fe700 1 -- 172.21.15.53:0/3730957652 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7fcd4000fd40 con 0x7fcd44100740 2023-07-14T19:18:51.227 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:50 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:50.969+0000 7ff46f1c8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:18:51.228 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:51.101+0000 7ff46f1c8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:18:51.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-mon[98015]: Standby manager daemon z started 2023-07-14T19:18:51.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-mon[98015]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T19:18:51.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-mon[98015]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:18:51.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-mon[98015]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T19:18:51.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-mon[98015]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:18:51.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1084158279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:18:51.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:51 smithi104 ceph-mon[98435]: Standby manager daemon z started 2023-07-14T19:18:51.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:51 smithi104 ceph-mon[98435]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T19:18:51.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:51 smithi104 ceph-mon[98435]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:18:51.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:51 smithi104 ceph-mon[98435]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T19:18:51.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:51 smithi104 ceph-mon[98435]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:18:51.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:51 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1084158279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:18:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:51 smithi053 ceph-mon[96565]: Standby manager daemon z started 2023-07-14T19:18:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:51 smithi053 ceph-mon[96565]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T19:18:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:51 smithi053 ceph-mon[96565]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:18:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:51 smithi053 ceph-mon[96565]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T19:18:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:51 smithi053 ceph-mon[96565]: from='mgr.? 172.21.15.104:0/2553678191' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:18:51.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:51 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1084158279' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:18:51.618 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:51.292+0000 7ff46f1c8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:18:51.619 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:51.404+0000 7ff46f1c8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:18:51.978 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:51 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:51.616+0000 7ff46f1c8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:18:52.478 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:18:52 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y[99496]: 2023-07-14T19:18:52.463+0000 7ff46f1c8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:18:52.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:52 smithi070 ceph-mon[98015]: mgrmap e14: x(active, since 69s), standbys: z 2023-07-14T19:18:52.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T19:18:52.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:52 smithi070 ceph-mon[98015]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:52.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:52 smithi104 ceph-mon[98435]: mgrmap e14: x(active, since 69s), standbys: z 2023-07-14T19:18:52.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T19:18:52.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:52 smithi104 ceph-mon[98435]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:52.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:52 smithi053 ceph-mon[96565]: mgrmap e14: x(active, since 69s), standbys: z 2023-07-14T19:18:52.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T19:18:52.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:52 smithi053 ceph-mon[96565]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:18:53.238+0000 7fcd3b7fe700 1 -- 172.21.15.53:0/3730957652 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fcd4000e3b0 con 0x7fcd44100740 2023-07-14T19:18:53.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:53 smithi104 ceph-mon[98435]: Standby manager daemon y started 2023-07-14T19:18:53.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:53 smithi104 ceph-mon[98435]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T19:18:53.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:53 smithi104 ceph-mon[98435]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:18:53.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:53 smithi104 ceph-mon[98435]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T19:18:53.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:53 smithi104 ceph-mon[98435]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:18:53.554 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:53 smithi053 ceph-mon[96565]: Standby manager daemon y started 2023-07-14T19:18:53.554 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:53 smithi053 ceph-mon[96565]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T19:18:53.554 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:53 smithi053 ceph-mon[96565]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:18:53.555 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:53 smithi053 ceph-mon[96565]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T19:18:53.555 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:53 smithi053 ceph-mon[96565]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:18:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:53 smithi070 ceph-mon[98015]: Standby manager daemon y started 2023-07-14T19:18:53.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:53 smithi070 ceph-mon[98015]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T19:18:53.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:53 smithi070 ceph-mon[98015]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:18:53.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:53 smithi070 ceph-mon[98015]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T19:18:53.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:53 smithi070 ceph-mon[98015]: from='mgr.? 172.21.15.70:0/1019420614' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:18:54.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:54 smithi104 ceph-mon[98435]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-07-14T19:18:54.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T19:18:54.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:54 smithi104 ceph-mon[98435]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:54.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:54 smithi053 ceph-mon[96565]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-07-14T19:18:54.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T19:18:54.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:54 smithi053 ceph-mon[96565]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:54.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:54 smithi070 ceph-mon[98015]: mgrmap e15: x(active, since 71s), standbys: z, y 2023-07-14T19:18:54.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T19:18:54.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:54 smithi070 ceph-mon[98015]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:55.481 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:55 smithi053 ceph-mon[96565]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:55.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:55 smithi070 ceph-mon[98015]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:55.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:55 smithi104 ceph-mon[98435]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:56.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:56 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T19:18:56.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:56 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:56.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:56 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T19:18:56.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:56 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:56.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:56 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T19:18:56.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:56 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:18:58.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:58 smithi070 ceph-mon[98015]: Deploying daemon osd.0 on smithi053 2023-07-14T19:18:58.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:58 smithi070 ceph-mon[98015]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:58.504 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:58 smithi053 ceph-mon[96565]: Deploying daemon osd.0 on smithi053 2023-07-14T19:18:58.505 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:58 smithi053 ceph-mon[96565]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:58.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:58 smithi104 ceph-mon[98435]: Deploying daemon osd.0 on smithi053 2023-07-14T19:18:58.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:58 smithi104 ceph-mon[98435]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:59.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:18:59 smithi070 ceph-mon[98015]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:59.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:18:59 smithi104 ceph-mon[98435]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:18:59.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:18:59 smithi053 ceph-mon[96565]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:01.957 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:01 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:01.957 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:01 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:01.957 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:01 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:01.957 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:01 smithi053 ceph-mon[96565]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:02.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:01 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:02.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:01 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:02.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:01 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:02.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:01 smithi070 ceph-mon[98015]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:02.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:01 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:02.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:01 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:02.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:01 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:02.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:01 smithi104 ceph-mon[98435]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:03.592 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.590+0000 7fcd3b7fe700 1 -- 172.21.15.53:0/3730957652 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcd44102300 con 0x7fcd300389e0 2023-07-14T19:19:03.593 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 0 on host 'smithi053' 2023-07-14T19:19:03.594 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcd300389e0 msgr2=0x7fcd3003ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:03.594 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcd300389e0 0x7fcd3003ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400b2e0 tx=0x7fcd34005fb0 comp rx=0 tx=0).stop 2023-07-14T19:19:03.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44100740 msgr2=0x7fcd441000a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:03.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44100740 0x7fcd441000a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000d980 tx=0x7fcd4000dc90 comp rx=0 tx=0).stop 2023-07-14T19:19:03.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 shutdown_connections 2023-07-14T19:19:03.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcd300389e0 0x7fcd3003ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:03.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd44105980 0x7fcd440fec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:03.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd44100740 0x7fcd441000a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:03.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 --2- 172.21.15.53:0/3730957652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd44103060 0x7fcd440fe6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:03.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 >> 172.21.15.53:0/3730957652 conn(0x7fcd440fa1e0 msgr2=0x7fcd44101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:19:03.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 shutdown_connections 2023-07-14T19:19:03.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:03.593+0000 7fcd4c504700 1 -- 172.21.15.53:0/3730957652 wait complete. 2023-07-14T19:19:04.051 DEBUG:teuthology.orchestra.run.smithi053:osd.0> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.0.service 2023-07-14T19:19:04.053 INFO:tasks.cephadm:Deploying osd.1 on smithi070 with /dev/vg_nvme/lv_4... 2023-07-14T19:19:04.054 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T19:19:04.090 INFO:journalctl@ceph.osd.0.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:19:04.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:03 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:03 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:03 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:04.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:03 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:04.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:03 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:03 smithi070 ceph-mon[98015]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:04.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:03 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:03 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:03 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:04.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:03 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:03 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:03 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:04.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:03 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:04.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:03 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:03 smithi104 ceph-mon[98435]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:04.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:03 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:03 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:03 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:04.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:03 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:03 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:03 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:04.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:03 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:04.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:03 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:03 smithi053 ceph-mon[96565]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:04.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:03 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:03 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:04.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:03 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:19:05.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:04 smithi053 ceph-mon[96565]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T19:19:05.072 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:04 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-0[109751]: 2023-07-14T19:19:04.811+0000 7fa2940bd640 -1 osd.0 0 log_to_monitors true 2023-07-14T19:19:05.104 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:04 smithi070 ceph-mon[98015]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T19:19:05.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:04 smithi104 ceph-mon[98435]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T19:19:05.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:05 smithi070 ceph-mon[98015]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T19:19:05.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:05 smithi070 ceph-mon[98015]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T19:19:05.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:05 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:05.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:05 smithi070 ceph-mon[98015]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:19:05.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:05 smithi070 ceph-mon[98015]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:06.182 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T19:19:06.183 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T19:19:06.183 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2023-07-14T19:19:06.183 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2023-07-14T19:19:06.184 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00834797 s, 1.3 GB/s 2023-07-14T19:19:06.184 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2023-07-14T19:19:06.257 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:05 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-0[109751]: 2023-07-14T19:19:05.967+0000 7fa288f34700 -1 osd.0 0 waiting for initial osdmap 2023-07-14T19:19:06.258 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:05 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-0[109751]: 2023-07-14T19:19:05.970+0000 7fa283d28700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:19:06.258 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:05 smithi053 ceph-mon[96565]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T19:19:06.258 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:05 smithi053 ceph-mon[96565]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T19:19:06.258 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:05 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:06.259 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:05 smithi053 ceph-mon[96565]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:19:06.259 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:05 smithi053 ceph-mon[96565]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:06.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:05 smithi104 ceph-mon[98435]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T19:19:06.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:05 smithi104 ceph-mon[98435]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T19:19:06.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:05 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:06.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:05 smithi104 ceph-mon[98435]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:19:06.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:05 smithi104 ceph-mon[98435]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:06.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi070:vg_nvme/lv_4 2023-07-14T19:19:07.198 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:19:07.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:19:07.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T19:19:07.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:07.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:07.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:19:07.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:07.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:07.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:19:07.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T19:19:07.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:07.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:07.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:19:07.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:07.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:07.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:19:07.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T19:19:07.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:07.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:07.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:19:07.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:07.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:07.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:07.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:08.160 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:07 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:19:08.161 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:07 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:19:08.161 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:07 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi053 2023-07-14T19:19:08.162 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:07 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi053 to 17173M 2023-07-14T19:19:08.162 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:07 smithi070 ceph-mon[98015]: osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] boot 2023-07-14T19:19:08.162 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:07 smithi070 ceph-mon[98015]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T19:19:08.162 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:07 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:08.163 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:07 smithi070 ceph-mon[98015]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:08.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:07 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:19:08.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:07 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:19:08.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:07 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi053 2023-07-14T19:19:08.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:07 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi053 to 17173M 2023-07-14T19:19:08.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:07 smithi104 ceph-mon[98435]: osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] boot 2023-07-14T19:19:08.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:07 smithi104 ceph-mon[98435]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T19:19:08.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:07 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:08.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:07 smithi104 ceph-mon[98435]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:08.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:07 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:19:08.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:07 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:19:08.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:07 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi053 2023-07-14T19:19:08.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:07 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi053 to 17173M 2023-07-14T19:19:08.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:07 smithi053 ceph-mon[96565]: osd.0 [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] boot 2023-07-14T19:19:08.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:07 smithi053 ceph-mon[96565]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T19:19:08.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:19:08.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:07 smithi053 ceph-mon[96565]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:19:08.356 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.354+0000 7fbf91060700 1 -- 172.21.15.70:0/1048722384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c0febd0 msgr2=0x7fbf8c0fefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:08.357 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.354+0000 7fbf91060700 1 --2- 172.21.15.70:0/1048722384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c0febd0 0x7fbf8c0fefe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009990 tx=0x7fbf80009ca0 comp rx=0 tx=0).stop 2023-07-14T19:19:08.357 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.354+0000 7fbf91060700 1 -- 172.21.15.70:0/1048722384 shutdown_connections 2023-07-14T19:19:08.357 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.354+0000 7fbf91060700 1 --2- 172.21.15.70:0/1048722384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c100820 0x7fbf8c1070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:08.357 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.354+0000 7fbf91060700 1 --2- 172.21.15.70:0/1048722384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c0febd0 0x7fbf8c0fefe0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:08.357 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.354+0000 7fbf91060700 1 --2- 172.21.15.70:0/1048722384 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf8c0ffe70 0x7fbf8c1002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:08.357 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.354+0000 7fbf91060700 1 -- 172.21.15.70:0/1048722384 >> 172.21.15.70:0/1048722384 conn(0x7fbf8c0fa240 msgr2=0x7fbf8c0fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:19:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.355+0000 7fbf91060700 1 -- 172.21.15.70:0/1048722384 shutdown_connections 2023-07-14T19:19:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.355+0000 7fbf91060700 1 -- 172.21.15.70:0/1048722384 wait complete. 2023-07-14T19:19:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.355+0000 7fbf91060700 1 Processor -- start 2023-07-14T19:19:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.355+0000 7fbf91060700 1 -- start start 2023-07-14T19:19:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf91060700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c0febd0 0x7fbf8c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf91060700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf8c0ffe70 0x7fbf8c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf91060700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c100820 0x7fbf8c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf91060700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c19d500 con 0x7fbf8c0febd0 2023-07-14T19:19:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf91060700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c19d670 con 0x7fbf8c100820 2023-07-14T19:19:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf91060700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c19d830 con 0x7fbf8c0ffe70 2023-07-14T19:19:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8b59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c100820 0x7fbf8c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8b59e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c100820 0x7fbf8c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56238/0 (socket says 172.21.15.70:56238) 2023-07-14T19:19:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8b59e700 1 -- 172.21.15.70:0/3315295609 learned_addr learned my addr 172.21.15.70:0/3315295609 (peer_addr_for_me v2:172.21.15.70:0/0) 2023-07-14T19:19:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8ad9d700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c0febd0 0x7fbf8c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8a59c700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf8c0ffe70 0x7fbf8c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8b59e700 1 -- 172.21.15.70:0/3315295609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf8c0ffe70 msgr2=0x7fbf8c19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8b59e700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf8c0ffe70 0x7fbf8c19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8b59e700 1 -- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c0febd0 msgr2=0x7fbf8c0695e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8b59e700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c0febd0 0x7fbf8c0695e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8b59e700 1 -- 172.21.15.70:0/3315295609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf800095f0 con 0x7fbf8c100820 2023-07-14T19:19:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf8b59e700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c100820 0x7fbf8c19cd60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400b970 tx=0x7fbf8400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:19:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf73fff700 1 -- 172.21.15.70:0/3315295609 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8400d860 con 0x7fbf8c100820 2023-07-14T19:19:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf73fff700 1 -- 172.21.15.70:0/3315295609 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf8400de80 con 0x7fbf8c100820 2023-07-14T19:19:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8c19db10 con 0x7fbf8c100820 2023-07-14T19:19:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.356+0000 7fbf73fff700 1 -- 172.21.15.70:0/3315295609 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84017400 con 0x7fbf8c100820 2023-07-14T19:19:08.365 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.357+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8c19dfc0 con 0x7fbf8c100820 2023-07-14T19:19:08.365 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.358+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8c106160 con 0x7fbf8c100820 2023-07-14T19:19:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.359+0000 7fbf73fff700 1 -- 172.21.15.70:0/3315295609 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fbf8400e630 con 0x7fbf8c100820 2023-07-14T19:19:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.359+0000 7fbf73fff700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbf740a10c0 0x7fbf740a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.359+0000 7fbf73fff700 1 -- 172.21.15.70:0/3315295609 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fbf84006e40 con 0x7fbf8c100820 2023-07-14T19:19:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.360+0000 7fbf8ad9d700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbf740a10c0 0x7fbf740a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.360+0000 7fbf8ad9d700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbf740a10c0 0x7fbf740a3570 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000b5c0 tx=0x7fbf80005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:19:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.364+0000 7fbf73fff700 1 -- 172.21.15.70:0/3315295609 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbf84095c00 con 0x7fbf8c100820 2023-07-14T19:19:08.560 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:08.559+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf8c060a80 con 0x7fbf740a10c0 2023-07-14T19:19:09.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:09 smithi070 ceph-mon[98015]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T19:19:09.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:09 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:19:09.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:09 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:19:09.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:09 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:09.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:09 smithi104 ceph-mon[98435]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T19:19:09.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:09 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:19:09.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:09 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:19:09.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:09 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:09.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:09 smithi053 ceph-mon[96565]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T19:19:09.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:19:09.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:19:09.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:10.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:10 smithi070 ceph-mon[98015]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:19:10.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:10 smithi070 ceph-mon[98015]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:10.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:10 smithi104 ceph-mon[98435]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:19:10.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:10 smithi104 ceph-mon[98435]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:10.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:10 smithi053 ceph-mon[96565]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:19:10.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:10 smithi053 ceph-mon[96565]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:11.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:11 smithi104 ceph-mon[98435]: from='client.? 172.21.15.70:0/1988671987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "301f0944-3cad-4319-9242-4a7e4c4bcc94"}]: dispatch 2023-07-14T19:19:11.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:11 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "301f0944-3cad-4319-9242-4a7e4c4bcc94"}]: dispatch 2023-07-14T19:19:11.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:11 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "301f0944-3cad-4319-9242-4a7e4c4bcc94"}]': finished 2023-07-14T19:19:11.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:11 smithi104 ceph-mon[98435]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T19:19:11.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:11.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:11 smithi053 ceph-mon[96565]: from='client.? 172.21.15.70:0/1988671987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "301f0944-3cad-4319-9242-4a7e4c4bcc94"}]: dispatch 2023-07-14T19:19:11.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:11 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "301f0944-3cad-4319-9242-4a7e4c4bcc94"}]: dispatch 2023-07-14T19:19:11.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:11 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "301f0944-3cad-4319-9242-4a7e4c4bcc94"}]': finished 2023-07-14T19:19:11.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:11 smithi053 ceph-mon[96565]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T19:19:11.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:11.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:11 smithi070 ceph-mon[98015]: from='client.? 172.21.15.70:0/1988671987' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "301f0944-3cad-4319-9242-4a7e4c4bcc94"}]: dispatch 2023-07-14T19:19:11.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:11 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "301f0944-3cad-4319-9242-4a7e4c4bcc94"}]: dispatch 2023-07-14T19:19:11.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:11 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "301f0944-3cad-4319-9242-4a7e4c4bcc94"}]': finished 2023-07-14T19:19:11.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:11 smithi070 ceph-mon[98015]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T19:19:11.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:12.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:12 smithi104 ceph-mon[98435]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:12.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:12 smithi104 ceph-mon[98435]: from='client.? 172.21.15.70:0/3926292172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:19:12.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:12 smithi053 ceph-mon[96565]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:12.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:12 smithi053 ceph-mon[96565]: from='client.? 172.21.15.70:0/3926292172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:19:12.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:12 smithi070 ceph-mon[98015]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:12.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:12 smithi070 ceph-mon[98015]: from='client.? 172.21.15.70:0/3926292172' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:19:13.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:13 smithi070 ceph-mon[98015]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:13.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:13 smithi104 ceph-mon[98435]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:13.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:13 smithi053 ceph-mon[96565]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:15.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:15 smithi070 ceph-mon[98015]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:15.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:15 smithi104 ceph-mon[98435]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:15.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:15 smithi053 ceph-mon[96565]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:18.351 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:18 smithi070 ceph-mon[98015]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:18.352 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:18 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T19:19:18.352 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:18 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:18.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:18 smithi104 ceph-mon[98435]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:18.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:18 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T19:19:18.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:18 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:18.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:18 smithi053 ceph-mon[96565]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:18.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:18 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T19:19:18.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:18 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:19.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:19 smithi070 ceph-mon[98015]: Deploying daemon osd.1 on smithi070 2023-07-14T19:19:19.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:19 smithi104 ceph-mon[98435]: Deploying daemon osd.1 on smithi070 2023-07-14T19:19:19.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:19 smithi053 ceph-mon[96565]: Deploying daemon osd.1 on smithi070 2023-07-14T19:19:20.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:20 smithi104 ceph-mon[98435]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:20.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:20 smithi053 ceph-mon[96565]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:20.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:20 smithi070 ceph-mon[98015]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:21.587 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:21 smithi070 ceph-mon[98015]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:21.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:21 smithi104 ceph-mon[98435]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:21.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:21 smithi053 ceph-mon[96565]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:23.167 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:22 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:23.167 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:22 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:23.167 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:22 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:23.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:22 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:23.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:22 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:23.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:22 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:23.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:22 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:23.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:22 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:23.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:22 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:24.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:23 smithi070 ceph-mon[98015]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:24.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:23 smithi104 ceph-mon[98435]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:24.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:23 smithi053 ceph-mon[96565]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:24.778 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.775+0000 7fbf73fff700 1 -- 172.21.15.70:0/3315295609 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbf8c060a80 con 0x7fbf740a10c0 2023-07-14T19:19:24.779 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 1 on host 'smithi070' 2023-07-14T19:19:24.780 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbf740a10c0 msgr2=0x7fbf740a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:24.780 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbf740a10c0 0x7fbf740a3570 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000b5c0 tx=0x7fbf80005960 comp rx=0 tx=0).stop 2023-07-14T19:19:24.781 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c100820 msgr2=0x7fbf8c19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:24.781 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c100820 0x7fbf8c19cd60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400b970 tx=0x7fbf8400bc80 comp rx=0 tx=0).stop 2023-07-14T19:19:24.781 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 shutdown_connections 2023-07-14T19:19:24.781 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbf740a10c0 0x7fbf740a3570 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:24.781 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf8c100820 0x7fbf8c19cd60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:24.782 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c0febd0 0x7fbf8c0695e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:24.782 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 --2- 172.21.15.70:0/3315295609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf8c0ffe70 0x7fbf8c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:24.782 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 >> 172.21.15.70:0/3315295609 conn(0x7fbf8c0fa240 msgr2=0x7fbf8c1032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:19:24.782 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 shutdown_connections 2023-07-14T19:19:24.782 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:19:24.777+0000 7fbf91060700 1 -- 172.21.15.70:0/3315295609 wait complete. 2023-07-14T19:19:25.184 DEBUG:teuthology.orchestra.run.smithi070:osd.1> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.1.service 2023-07-14T19:19:25.186 INFO:tasks.cephadm:Deploying osd.2 on smithi104 with /dev/vg_nvme/lv_4... 2023-07-14T19:19:25.186 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T19:19:25.211 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.211 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.211 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:25.211 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:25.212 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.212 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.212 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.212 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:25.216 INFO:journalctl@ceph.osd.1.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:19:25.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:25.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:25.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:25.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:25.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:25.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:25.749 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:19:26.131 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:25 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-1[103115]: 2023-07-14T19:19:25.873+0000 7efd75861640 -1 osd.1 0 log_to_monitors true 2023-07-14T19:19:26.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:26 smithi104 ceph-mon[98435]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:26.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:26 smithi104 ceph-mon[98435]: from='osd.1 [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:19:26.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:26 smithi104 ceph-mon[98435]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:19:26.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:26 smithi070 ceph-mon[98015]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:26.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:26 smithi070 ceph-mon[98015]: from='osd.1 [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:19:26.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:26 smithi070 ceph-mon[98015]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:19:26.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:26 smithi053 ceph-mon[96565]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:26.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:26 smithi053 ceph-mon[96565]: from='osd.1 [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:19:26.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:26 smithi053 ceph-mon[96565]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:19:27.335 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T19:19:27.335 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T19:19:27.336 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2023-07-14T19:19:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2023-07-14T19:19:27.336 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111335 s, 942 MB/s 2023-07-14T19:19:27.336 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2023-07-14T19:19:27.478 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:27 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-1[103115]: 2023-07-14T19:19:27.142+0000 7efd6a6d8700 -1 osd.1 0 waiting for initial osdmap 2023-07-14T19:19:27.478 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:27 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-1[103115]: 2023-07-14T19:19:27.145+0000 7efd644ca700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:19:27.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:27 smithi070 ceph-mon[98015]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T19:19:27.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:27 smithi070 ceph-mon[98015]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T19:19:27.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:27.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:27 smithi070 ceph-mon[98015]: from='osd.1 [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:19:27.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:27 smithi070 ceph-mon[98015]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:19:27.487 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:27 smithi104 ceph-mon[98435]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T19:19:27.488 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:27 smithi104 ceph-mon[98435]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T19:19:27.488 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:27.488 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:27 smithi104 ceph-mon[98435]: from='osd.1 [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:19:27.488 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:27 smithi104 ceph-mon[98435]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:19:27.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:27 smithi053 ceph-mon[96565]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T19:19:27.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:27 smithi053 ceph-mon[96565]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T19:19:27.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:27.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:27 smithi053 ceph-mon[96565]: from='osd.1 [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:19:27.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:27 smithi053 ceph-mon[96565]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:19:27.804 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2023-07-14T19:19:28.354 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:19:28.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:19:28.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T19:19:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi070 2023-07-14T19:19:28.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:19:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi070 to 17168M 2023-07-14T19:19:28.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:28 smithi070 ceph-mon[98015]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:28.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:19:28.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T19:19:28.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:28.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:28.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi070 2023-07-14T19:19:28.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:19:28.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi070 to 17168M 2023-07-14T19:19:28.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:28.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:28.514 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.514 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:28 smithi104 ceph-mon[98435]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:28.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:19:28.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T19:19:28.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:28.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:28.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi070 2023-07-14T19:19:28.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:19:28.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi070 to 17168M 2023-07-14T19:19:28.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:28.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:28.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:28.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:28 smithi053 ceph-mon[96565]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:19:29.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:29 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:19:29.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:29 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:19:29.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:29 smithi104 ceph-mon[98435]: osd.1 [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] boot 2023-07-14T19:19:29.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:29 smithi104 ceph-mon[98435]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T19:19:29.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:29.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:29 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:19:29.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:29 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:19:29.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:29 smithi070 ceph-mon[98015]: osd.1 [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] boot 2023-07-14T19:19:29.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:29 smithi070 ceph-mon[98015]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T19:19:29.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:29.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:29 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:19:29.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:29 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:19:29.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:29 smithi053 ceph-mon[96565]: osd.1 [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] boot 2023-07-14T19:19:29.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:29 smithi053 ceph-mon[96565]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T19:19:29.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:19:29.590 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 -- 172.21.15.104:0/1811635271 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6244100740 msgr2=0x7f6244107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 --2- 172.21.15.104:0/1811635271 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6244100740 0x7f6244107000 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f623c00b0a0 tx=0x7f623c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:19:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 -- 172.21.15.104:0/1811635271 shutdown_connections 2023-07-14T19:19:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 --2- 172.21.15.104:0/1811635271 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6244100740 0x7f6244107000 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 --2- 172.21.15.104:0/1811635271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62440febb0 0x7f62440fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 --2- 172.21.15.104:0/1811635271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62440ffdb0 0x7f6244100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 -- 172.21.15.104:0/1811635271 >> 172.21.15.104:0/1811635271 conn(0x7f62440fa1e0 msgr2=0x7f62440fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:19:29.591 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 -- 172.21.15.104:0/1811635271 shutdown_connections 2023-07-14T19:19:29.592 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 -- 172.21.15.104:0/1811635271 wait complete. 2023-07-14T19:19:29.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.588+0000 7f624908b700 1 Processor -- start 2023-07-14T19:19:29.593 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624908b700 1 -- start start 2023-07-14T19:19:29.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624908b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62440febb0 0x7f6244068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:29.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624908b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62440ffdb0 0x7f624419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:29.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624908b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6244100740 0x7f624419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:29.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624908b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624419d680 con 0x7f62440febb0 2023-07-14T19:19:29.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624908b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624419d7f0 con 0x7f6244100740 2023-07-14T19:19:29.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624908b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624419d9b0 con 0x7f62440ffdb0 2023-07-14T19:19:29.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62440ffdb0 0x7f624419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:29.594 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62440ffdb0 0x7f624419a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56908/0 (socket says 172.21.15.104:56908) 2023-07-14T19:19:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624259c700 1 -- 172.21.15.104:0/151129607 learned_addr learned my addr 172.21.15.104:0/151129607 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:19:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624359e700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6244100740 0x7f624419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f6242d9d700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62440febb0 0x7f6244068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624259c700 1 -- 172.21.15.104:0/151129607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6244100740 msgr2=0x7f624419cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624259c700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6244100740 0x7f624419cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624259c700 1 -- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62440febb0 msgr2=0x7f6244068c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:29.595 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624259c700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62440febb0 0x7f6244068c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:29.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624259c700 1 -- 172.21.15.104:0/151129607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c009d00 con 0x7f62440ffdb0 2023-07-14T19:19:29.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f6242d9d700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62440febb0 0x7f6244068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:19:29.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.589+0000 7f624259c700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62440ffdb0 0x7f624419a8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f623400c970 tx=0x7f623400cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:19:29.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.590+0000 7f622bfff700 1 -- 172.21.15.104:0/151129607 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234007aa0 con 0x7f62440ffdb0 2023-07-14T19:19:29.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.590+0000 7f622bfff700 1 -- 172.21.15.104:0/151129607 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6234004cf0 con 0x7f62440ffdb0 2023-07-14T19:19:29.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.590+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624419dc90 con 0x7f62440ffdb0 2023-07-14T19:19:29.596 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.590+0000 7f622bfff700 1 -- 172.21.15.104:0/151129607 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6234005600 con 0x7f62440ffdb0 2023-07-14T19:19:29.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.590+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f624419e140 con 0x7f62440ffdb0 2023-07-14T19:19:29.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.591+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6244106130 con 0x7f62440ffdb0 2023-07-14T19:19:29.597 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.592+0000 7f622bfff700 1 -- 172.21.15.104:0/151129607 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f6234004750 con 0x7f62440ffdb0 2023-07-14T19:19:29.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.592+0000 7f622bfff700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f622c0a10c0 0x7f622c0a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:29.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.592+0000 7f622bfff700 1 -- 172.21.15.104:0/151129607 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f6234013070 con 0x7f62440ffdb0 2023-07-14T19:19:29.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.592+0000 7f6242d9d700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f622c0a10c0 0x7f622c0a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:29.598 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.593+0000 7f6242d9d700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f622c0a10c0 0x7f622c0a3570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f62440ffc10 tx=0x7f6238008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:19:29.599 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.596+0000 7f622bfff700 1 -- 172.21.15.104:0/151129607 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6234095e90 con 0x7f62440ffdb0 2023-07-14T19:19:29.791 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:29.788+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6244060a80 con 0x7f622c0a10c0 2023-07-14T19:19:30.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:30 smithi104 ceph-mon[98435]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T19:19:30.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:30 smithi104 ceph-mon[98435]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:30.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:19:30.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:19:30.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:30 smithi053 ceph-mon[96565]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T19:19:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:30 smithi053 ceph-mon[96565]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:19:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:19:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:30.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:30 smithi070 ceph-mon[98015]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T19:19:30.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:30 smithi070 ceph-mon[98015]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:30.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:19:30.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:19:30.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:31.485 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:31 smithi104 ceph-mon[98435]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:19:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:31 smithi053 ceph-mon[96565]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:19:31.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:31 smithi070 ceph-mon[98015]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:19:32.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:32 smithi104 ceph-mon[98435]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:32 smithi053 ceph-mon[96565]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:32 smithi070 ceph-mon[98015]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:33 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/1856570739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447"}]: dispatch 2023-07-14T19:19:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:33 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447"}]: dispatch 2023-07-14T19:19:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:33 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447"}]': finished 2023-07-14T19:19:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:33 smithi053 ceph-mon[96565]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T19:19:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:33.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:33 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/767336208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:19:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:33 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/1856570739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447"}]: dispatch 2023-07-14T19:19:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:33 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447"}]: dispatch 2023-07-14T19:19:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:33 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447"}]': finished 2023-07-14T19:19:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:33 smithi070 ceph-mon[98015]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T19:19:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:33 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:33.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:33 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/767336208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:19:33.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:33 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/1856570739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447"}]: dispatch 2023-07-14T19:19:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:33 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447"}]: dispatch 2023-07-14T19:19:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:33 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447"}]': finished 2023-07-14T19:19:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:33 smithi104 ceph-mon[98435]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T19:19:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:33.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:33 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/767336208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:19:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:34 smithi053 ceph-mon[96565]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:34.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:34 smithi070 ceph-mon[98015]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:34.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:34 smithi104 ceph-mon[98435]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:36.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:36 smithi104 ceph-mon[98435]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:36 smithi053 ceph-mon[96565]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:36 smithi070 ceph-mon[98015]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:38.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:38 smithi070 ceph-mon[98015]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:38.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:38 smithi104 ceph-mon[98435]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:38 smithi053 ceph-mon[96565]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:39.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T19:19:39.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:39.498 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T19:19:39.498 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T19:19:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:40.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:40 smithi104 ceph-mon[98435]: Deploying daemon osd.2 on smithi104 2023-07-14T19:19:40.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:40 smithi104 ceph-mon[98435]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:40.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:40 smithi053 ceph-mon[96565]: Deploying daemon osd.2 on smithi104 2023-07-14T19:19:40.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:40 smithi053 ceph-mon[96565]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:40.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:40 smithi070 ceph-mon[98015]: Deploying daemon osd.2 on smithi104 2023-07-14T19:19:40.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:40 smithi070 ceph-mon[98015]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:41 smithi070 ceph-mon[98015]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:41 smithi104 ceph-mon[98435]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:41.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:41 smithi053 ceph-mon[96565]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:44.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:44 smithi104 ceph-mon[98435]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:44.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:44.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:44.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:44 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:44 smithi053 ceph-mon[96565]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:44 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:44.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:44 smithi070 ceph-mon[98015]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:44.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:44.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:44.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:44 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:46.307 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.305+0000 7f622bfff700 1 -- 172.21.15.104:0/151129607 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6244060a80 con 0x7f622c0a10c0 2023-07-14T19:19:46.308 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 2 on host 'smithi104' 2023-07-14T19:19:46.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f622c0a10c0 msgr2=0x7f622c0a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:46.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f622c0a10c0 0x7f622c0a3570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f62440ffc10 tx=0x7f6238008040 comp rx=0 tx=0).stop 2023-07-14T19:19:46.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62440ffdb0 msgr2=0x7f624419a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:46.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62440ffdb0 0x7f624419a8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f623400c970 tx=0x7f623400cd30 comp rx=0 tx=0).stop 2023-07-14T19:19:46.309 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 shutdown_connections 2023-07-14T19:19:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f622c0a10c0 0x7f622c0a3570 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6244100740 0x7f624419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62440febb0 0x7f6244068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 --2- 172.21.15.104:0/151129607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62440ffdb0 0x7f624419a8a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.307+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 >> 172.21.15.104:0/151129607 conn(0x7f62440fa1e0 msgr2=0x7f6244103270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:19:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.308+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 shutdown_connections 2023-07-14T19:19:46.310 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:19:46.308+0000 7f624908b700 1 -- 172.21.15.104:0/151129607 wait complete. 2023-07-14T19:19:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:46 smithi053 ceph-mon[96565]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:46.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:46.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.720 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:46 smithi104 ceph-mon[98435]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:46.720 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.720 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.720 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:46.721 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:46.721 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.721 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.721 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:46 smithi070 ceph-mon[98015]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:46.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:46.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:46.748 DEBUG:teuthology.orchestra.run.smithi104:osd.2> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.2.service 2023-07-14T19:19:46.751 INFO:tasks.cephadm:Deploying osd.3 on smithi053 with /dev/vg_nvme/lv_3... 2023-07-14T19:19:46.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T19:19:46.782 INFO:journalctl@ceph.osd.2.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:19:47.328 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:19:47.511 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:47 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-2[104321]: 2023-07-14T19:19:47.243+0000 7f9d30109640 -1 osd.2 0 log_to_monitors true 2023-07-14T19:19:47.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:47 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:47.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:47 smithi104 ceph-mon[98435]: from='osd.2 [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:19:47.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:47 smithi104 ceph-mon[98435]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:19:47.540 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:47 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:47.540 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:47 smithi053 ceph-mon[96565]: from='osd.2 [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:19:47.541 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:47 smithi053 ceph-mon[96565]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:19:47.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:47 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:19:47.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:47 smithi070 ceph-mon[98015]: from='osd.2 [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:19:47.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:47 smithi070 ceph-mon[98015]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:19:48.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:48 smithi104 ceph-mon[98435]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:48.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:48 smithi104 ceph-mon[98435]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T19:19:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:48 smithi104 ceph-mon[98435]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T19:19:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:48 smithi104 ceph-mon[98435]: from='osd.2 [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:19:48.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:48 smithi104 ceph-mon[98435]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:19:48.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:48 smithi053 ceph-mon[96565]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:48.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:48 smithi053 ceph-mon[96565]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T19:19:48.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:48 smithi053 ceph-mon[96565]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T19:19:48.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:48.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:48 smithi053 ceph-mon[96565]: from='osd.2 [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:19:48.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:48 smithi053 ceph-mon[96565]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:19:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:48 smithi070 ceph-mon[98015]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:48 smithi070 ceph-mon[98015]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T19:19:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:48 smithi070 ceph-mon[98015]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T19:19:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:48.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:48 smithi070 ceph-mon[98015]: from='osd.2 [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:19:48.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:48 smithi070 ceph-mon[98015]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:19:48.824 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T19:19:48.824 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T19:19:48.825 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2023-07-14T19:19:48.825 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2023-07-14T19:19:48.825 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00814885 s, 1.3 GB/s 2023-07-14T19:19:48.825 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2023-07-14T19:19:49.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi053:vg_nvme/lv_3 2023-07-14T19:19:49.511 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-2[104321]: 2023-07-14T19:19:49.233+0000 7f9d24f80700 -1 osd.2 0 waiting for initial osdmap 2023-07-14T19:19:49.511 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-2[104321]: 2023-07-14T19:19:49.238+0000 7f9d1f573700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:19:49.735 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:19:50.216 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi104 2023-07-14T19:19:50.217 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.217 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.217 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:19:50.218 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi104 to 17168M 2023-07-14T19:19:50.218 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.218 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:50.219 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:50.219 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.219 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:19:50.219 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T19:19:50.220 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:50.220 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:50.220 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:49 smithi053 ceph-mon[96565]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:50.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi104 2023-07-14T19:19:50.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:19:50.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi104 to 17168M 2023-07-14T19:19:50.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:50.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:50.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:19:50.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T19:19:50.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:50.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:50.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:49 smithi070 ceph-mon[98015]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:50.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi104 2023-07-14T19:19:50.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:19:50.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi104 to 17168M 2023-07-14T19:19:50.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:50.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:19:50.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:50.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:19:50.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T19:19:50.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:50.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:50.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:49 smithi104 ceph-mon[98435]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:19:51.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 -- 172.21.15.53:0/2722358656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a740febb0 msgr2=0x7f0a740fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:51.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/2722358656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a740febb0 0x7f0a740fefc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009990 tx=0x7f0a68009ca0 comp rx=0 tx=0).stop 2023-07-14T19:19:51.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 -- 172.21.15.53:0/2722358656 shutdown_connections 2023-07-14T19:19:51.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/2722358656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a74100740 0x7f0a74107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:51.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/2722358656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a740febb0 0x7f0a740fefc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:51.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/2722358656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a740ffdb0 0x7f0a74100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:51.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 -- 172.21.15.53:0/2722358656 >> 172.21.15.53:0/2722358656 conn(0x7f0a740fa1e0 msgr2=0x7f0a740fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:19:51.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 -- 172.21.15.53:0/2722358656 shutdown_connections 2023-07-14T19:19:51.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 -- 172.21.15.53:0/2722358656 wait complete. 2023-07-14T19:19:51.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 Processor -- start 2023-07-14T19:19:51.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.026+0000 7f0a7af5b700 1 -- start start 2023-07-14T19:19:51.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a7af5b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a740febb0 0x7f0a74068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:51.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a7af5b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a740ffdb0 0x7f0a7419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:51.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a7af5b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a74100740 0x7f0a7419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:51.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a7af5b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7419d480 con 0x7f0a74100740 2023-07-14T19:19:51.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a7af5b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7419d5f0 con 0x7f0a740febb0 2023-07-14T19:19:51.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a7af5b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7419d7b0 con 0x7f0a740ffdb0 2023-07-14T19:19:51.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a794f8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a74100740 0x7f0a7419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:51.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a794f8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a74100740 0x7f0a7419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49382/0 (socket says 172.21.15.53:49382) 2023-07-14T19:19:51.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a794f8700 1 -- 172.21.15.53:0/1758452215 learned_addr learned my addr 172.21.15.53:0/1758452215 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:19:51.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a78cf7700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a740febb0 0x7f0a74068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:51.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a73fff700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a740ffdb0 0x7f0a7419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:51.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a794f8700 1 -- 172.21.15.53:0/1758452215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a740ffdb0 msgr2=0x7f0a7419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:51.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a794f8700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a740ffdb0 0x7f0a7419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:51.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a794f8700 1 -- 172.21.15.53:0/1758452215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a740febb0 msgr2=0x7f0a74068c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:19:51.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a794f8700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a740febb0 0x7f0a74068c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:19:51.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a794f8700 1 -- 172.21.15.53:0/1758452215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a680095f0 con 0x7f0a74100740 2023-07-14T19:19:51.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a794f8700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a74100740 0x7f0a7419cce0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c007d70 tx=0x7f0a6c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:19:51.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a71ffb700 1 -- 172.21.15.53:0/1758452215 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c00ad40 con 0x7f0a74100740 2023-07-14T19:19:51.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a71ffb700 1 -- 172.21.15.53:0/1758452215 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a6c0128e0 con 0x7f0a74100740 2023-07-14T19:19:51.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a71ffb700 1 -- 172.21.15.53:0/1758452215 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0116b0 con 0x7f0a74100740 2023-07-14T19:19:51.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.027+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7419da90 con 0x7f0a74100740 2023-07-14T19:19:51.040 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.028+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7419df40 con 0x7f0a74100740 2023-07-14T19:19:51.040 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.029+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a74106150 con 0x7f0a74100740 2023-07-14T19:19:51.043 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.030+0000 7f0a71ffb700 1 -- 172.21.15.53:0/1758452215 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f0a6c012420 con 0x7f0a74100740 2023-07-14T19:19:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.030+0000 7f0a71ffb700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0a640a10c0 0x7f0a640a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:19:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.030+0000 7f0a78cf7700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0a640a10c0 0x7f0a640a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:19:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.030+0000 7f0a71ffb700 1 -- 172.21.15.53:0/1758452215 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f0a6c0cb530 con 0x7f0a74100740 2023-07-14T19:19:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.031+0000 7f0a78cf7700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0a640a10c0 0x7f0a640a3570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0a68006010 tx=0x7f0a68005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:19:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.035+0000 7f0a71ffb700 1 -- 172.21.15.53:0/1758452215 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0a6c0962b0 con 0x7f0a74100740 2023-07-14T19:19:51.245 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:51.243+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a74060a80 con 0x7f0a640a10c0 2023-07-14T19:19:51.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:51 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:19:51.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:51 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:19:51.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:51 smithi104 ceph-mon[98435]: osd.2 [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] boot 2023-07-14T19:19:51.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:51 smithi104 ceph-mon[98435]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T19:19:51.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:51 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:19:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:51 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:19:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:51 smithi053 ceph-mon[96565]: osd.2 [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] boot 2023-07-14T19:19:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:51 smithi053 ceph-mon[96565]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T19:19:51.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:51.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:51 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:19:51.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:51 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:19:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:51 smithi070 ceph-mon[98015]: osd.2 [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] boot 2023-07-14T19:19:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:51 smithi070 ceph-mon[98015]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T19:19:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:19:52.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:52 smithi053 ceph-mon[96565]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:19:52.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:19:52.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:19:52.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:52.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:52 smithi053 ceph-mon[96565]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:52.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:19:52.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:52 smithi104 ceph-mon[98435]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:19:52.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:19:52.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:19:52.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:52.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:52 smithi104 ceph-mon[98435]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:52.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:19:52.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:52 smithi070 ceph-mon[98015]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:19:52.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:19:52.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:19:52.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:19:52.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:52 smithi070 ceph-mon[98015]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:52.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:19:53.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T19:19:53.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:53 smithi104 ceph-mon[98435]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T19:19:53.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:19:53.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T19:19:53.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:53 smithi053 ceph-mon[96565]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T19:19:53.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:19:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T19:19:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:53 smithi070 ceph-mon[98015]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T19:19:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:19:54.072 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:53 smithi053 sudo[112894]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:19:54.072 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:53 smithi053 sudo[112894]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:54.072 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:53 smithi053 sudo[112894]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:54.072 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:53 smithi053 sudo[112894]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:54.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:19:54.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:54 smithi070 ceph-mon[98015]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T19:19:54.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:54 smithi070 ceph-mon[98015]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:54.505 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:54 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3256374398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b06af11-cdd7-426e-8f3c-68b2b33e750a"}]: dispatch 2023-07-14T19:19:54.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:54 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b06af11-cdd7-426e-8f3c-68b2b33e750a"}]: dispatch 2023-07-14T19:19:54.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:54 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b06af11-cdd7-426e-8f3c-68b2b33e750a"}]': finished 2023-07-14T19:19:54.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:54 smithi070 ceph-mon[98015]: osdmap e21: 4 total, 3 up, 4 in 2023-07-14T19:19:54.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:19:54.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:54 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/956880047' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:19:54.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:19:54.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:54 smithi104 ceph-mon[98435]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T19:19:54.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:54 smithi104 ceph-mon[98435]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:54.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:54 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3256374398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b06af11-cdd7-426e-8f3c-68b2b33e750a"}]: dispatch 2023-07-14T19:19:54.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:54 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b06af11-cdd7-426e-8f3c-68b2b33e750a"}]: dispatch 2023-07-14T19:19:54.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:54 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b06af11-cdd7-426e-8f3c-68b2b33e750a"}]': finished 2023-07-14T19:19:54.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:54 smithi104 ceph-mon[98435]: osdmap e21: 4 total, 3 up, 4 in 2023-07-14T19:19:54.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:19:54.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:54 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/956880047' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:19:54.572 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:54 smithi053 sudo[112927]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:19:54.572 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:54 smithi053 sudo[112927]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:54.572 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:54 smithi053 sudo[112927]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:54.572 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:19:54 smithi053 sudo[112927]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:54.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:19:54.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:54 smithi053 ceph-mon[96565]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T19:19:54.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:54 smithi053 ceph-mon[96565]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:54.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:54 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3256374398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b06af11-cdd7-426e-8f3c-68b2b33e750a"}]: dispatch 2023-07-14T19:19:54.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:54 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b06af11-cdd7-426e-8f3c-68b2b33e750a"}]: dispatch 2023-07-14T19:19:54.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:54 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b06af11-cdd7-426e-8f3c-68b2b33e750a"}]': finished 2023-07-14T19:19:54.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:54 smithi053 ceph-mon[96565]: osdmap e21: 4 total, 3 up, 4 in 2023-07-14T19:19:54.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:19:54.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:54 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/956880047' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:19:54.600 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:19:54.599+0000 7f0a71ffb700 1 -- 172.21.15.53:0/1758452215 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0a6c096680 con 0x7f0a74100740 2023-07-14T19:19:54.761 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:54 smithi070 sudo[105327]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:19:54.761 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:54 smithi070 sudo[105327]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:54.761 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:54 smithi070 sudo[105327]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:54.761 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:54 smithi070 sudo[105327]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:55.227 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:54 smithi070 sudo[105330]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:19:55.228 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:54 smithi070 sudo[105330]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:55.228 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:54 smithi070 sudo[105330]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:55.228 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:19:54 smithi070 sudo[105330]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:55.282 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:55 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:55.283 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:55 smithi104 ceph-mon[98435]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T19:19:55.283 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:55 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:55.283 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:55 smithi104 sudo[106493]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:19:55.283 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:55 smithi104 sudo[106493]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:55.284 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:55 smithi104 sudo[106493]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:55.284 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:55 smithi104 sudo[106493]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:55.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:55 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:55.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:55 smithi053 ceph-mon[96565]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T19:19:55.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:55 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:55.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:55 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:55.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:55 smithi070 ceph-mon[98015]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T19:19:55.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:55 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:55.760 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:55 smithi104 sudo[106496]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:19:55.761 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:55 smithi104 sudo[106496]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:55.761 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:55 smithi104 sudo[106496]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:55.761 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:19:55 smithi104 sudo[106496]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:56.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:55 smithi053 sudo[113544]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:19:56.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:55 smithi053 sudo[113544]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:56.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:55 smithi053 sudo[113544]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:56.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:55 smithi053 sudo[113544]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:56.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:56 smithi053 sudo[113734]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:19:56.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:56 smithi053 sudo[113734]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:56.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:56 smithi053 sudo[113734]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:56.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:56 smithi053 sudo[113734]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:56.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:56 smithi104 sudo[106499]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:19:56.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:56 smithi104 sudo[106499]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:56.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:56 smithi104 sudo[106499]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:56.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:56 smithi104 sudo[106499]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:57.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:56 smithi104 sudo[106502]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:19:57.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:56 smithi104 sudo[106502]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:57.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:56 smithi104 sudo[106502]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:57.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:56 smithi104 sudo[106502]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:57.193 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:56 smithi070 sudo[105333]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:19:57.193 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:56 smithi070 sudo[105333]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:57.193 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:56 smithi070 sudo[105333]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:57.193 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:56 smithi070 sudo[105333]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:57.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 sudo[105336]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:19:57.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 sudo[105336]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:19:57.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 sudo[105336]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:19:57.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 sudo[105336]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:19:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:19:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:19:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:19:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:19:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:19:57.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:19:57.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:19:57.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:57 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:19:57.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:57.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:57.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:19:57.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:19:57.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:19:57.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:19:57.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:19:57.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:19:57.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:19:57.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:57 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:19:57.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:57.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:19:57.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:19:57.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:19:57.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:19:57.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:19:57.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:19:57.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:19:57.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:19:57.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:19:58.456 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:58 smithi053 ceph-mon[96565]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:19:58.456 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:58 smithi053 ceph-mon[96565]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:19:58.457 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:58 smithi053 ceph-mon[96565]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:19:58.457 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:58 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:19:58.457 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:58 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:19:58.457 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:58 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:19:58.457 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:58 smithi053 ceph-mon[96565]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:19:58.458 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:58 smithi053 ceph-mon[96565]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:58.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:58 smithi070 ceph-mon[98015]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:19:58.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:58 smithi070 ceph-mon[98015]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:19:58.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:58 smithi070 ceph-mon[98015]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:19:58.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:58 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:19:58.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:58 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:19:58.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:58 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:19:58.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:58 smithi070 ceph-mon[98015]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:19:58.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:58 smithi070 ceph-mon[98015]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:58.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:58 smithi104 ceph-mon[98435]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:19:58.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:58 smithi104 ceph-mon[98435]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:19:58.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:58 smithi104 ceph-mon[98435]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:19:58.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:58 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:19:58.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:58 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:19:58.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:58 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:19:58.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:58 smithi104 ceph-mon[98435]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:19:58.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:58 smithi104 ceph-mon[98435]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:19:59.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:19:59 smithi053 ceph-mon[96565]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2023-07-14T19:19:59.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:19:59 smithi070 ceph-mon[98015]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2023-07-14T19:19:59.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:19:59 smithi104 ceph-mon[98435]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2023-07-14T19:20:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:00 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T19:20:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:00 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:00 smithi070 ceph-mon[98015]: Deploying daemon osd.3 on smithi053 2023-07-14T19:20:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:00 smithi070 ceph-mon[98015]: overall HEALTH_OK 2023-07-14T19:20:00.618 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:00 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T19:20:00.619 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:00 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:00.619 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:00 smithi053 ceph-mon[96565]: Deploying daemon osd.3 on smithi053 2023-07-14T19:20:00.619 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:00 smithi053 ceph-mon[96565]: overall HEALTH_OK 2023-07-14T19:20:00.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:00 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T19:20:00.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:00 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:00.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:00 smithi104 ceph-mon[98435]: Deploying daemon osd.3 on smithi053 2023-07-14T19:20:00.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:00 smithi104 ceph-mon[98435]: overall HEALTH_OK 2023-07-14T19:20:01.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:01 smithi070 ceph-mon[98015]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-07-14T19:20:01.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:01 smithi104 ceph-mon[98435]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-07-14T19:20:01.766 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:01 smithi053 ceph-mon[96565]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-07-14T19:20:03.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:03 smithi070 ceph-mon[98015]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-07-14T19:20:03.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:03 smithi104 ceph-mon[98435]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-07-14T19:20:03.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:03 smithi053 ceph-mon[96565]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-07-14T19:20:05.414 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:05 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:05.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:05 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:05.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:05 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:05.415 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:05 smithi053 ceph-mon[96565]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-07-14T19:20:05.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:05 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:05.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:05 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:05.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:05 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:05.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:05 smithi070 ceph-mon[98015]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-07-14T19:20:05.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:05 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:05.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:05 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:05.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:05 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:05.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:05 smithi104 ceph-mon[98435]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-07-14T19:20:06.995 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 3 on host 'smithi053' 2023-07-14T19:20:06.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.993+0000 7f0a71ffb700 1 -- 172.21.15.53:0/1758452215 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0a74060a80 con 0x7f0a640a10c0 2023-07-14T19:20:06.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0a640a10c0 msgr2=0x7f0a640a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:06.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0a640a10c0 0x7f0a640a3570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0a68006010 tx=0x7f0a68005960 comp rx=0 tx=0).stop 2023-07-14T19:20:06.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a74100740 msgr2=0x7f0a7419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:06.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a74100740 0x7f0a7419cce0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c007d70 tx=0x7f0a6c00a340 comp rx=0 tx=0).stop 2023-07-14T19:20:06.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 shutdown_connections 2023-07-14T19:20:06.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0a640a10c0 0x7f0a640a3570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:06.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a74100740 0x7f0a7419cce0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:06.999 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a740febb0 0x7f0a74068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:06.999 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 --2- 172.21.15.53:0/1758452215 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a740ffdb0 0x7f0a7419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:06.999 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 >> 172.21.15.53:0/1758452215 conn(0x7f0a740fa1e0 msgr2=0x7f0a74103290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:20:06.999 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 shutdown_connections 2023-07-14T19:20:06.999 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:06.996+0000 7f0a7af5b700 1 -- 172.21.15.53:0/1758452215 wait complete. 2023-07-14T19:20:07.417 DEBUG:teuthology.orchestra.run.smithi053:osd.3> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.3.service 2023-07-14T19:20:07.428 INFO:tasks.cephadm:Deploying osd.4 on smithi070 with /dev/vg_nvme/lv_3... 2023-07-14T19:20:07.429 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T19:20:07.456 INFO:journalctl@ceph.osd.3.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:20:07.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:07 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:07.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:07 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:07.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:07 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:07.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:07 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:07.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:07 smithi070 ceph-mon[98015]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-14T19:20:07.980 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:07.980 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:07.981 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:07.981 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:07.981 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:07 smithi053 ceph-mon[96565]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-14T19:20:07.994 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:20:08.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:07 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:08.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:07 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:08.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:07 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:08.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:07 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:08.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:07 smithi104 ceph-mon[98435]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-14T19:20:08.572 INFO:journalctl@ceph.osd.3.smithi053.stdout:Jul 14 19:20:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-3[115173]: 2023-07-14T19:20:08.281+0000 7f0d249b5640 -1 osd.3 0 log_to_monitors true 2023-07-14T19:20:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T19:20:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T19:20:09.443 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2023-07-14T19:20:09.444 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2023-07-14T19:20:09.444 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00822291 s, 1.3 GB/s 2023-07-14T19:20:09.444 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2023-07-14T19:20:09.497 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:09 smithi053 ceph-mon[96565]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T19:20:09.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:09 smithi104 ceph-mon[98435]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T19:20:09.574 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:09 smithi070 ceph-mon[98015]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T19:20:09.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi070:vg_nvme/lv_3 2023-07-14T19:20:10.390 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:20:10.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T19:20:10.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T19:20:10.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:10.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:20:10.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:20:10.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:10.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:10.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:10.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:10.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:10.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:10.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:10.514 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.572 INFO:journalctl@ceph.osd.3.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-3[115173]: 2023-07-14T19:20:10.240+0000 7f0d1982c700 -1 osd.3 0 waiting for initial osdmap 2023-07-14T19:20:10.573 INFO:journalctl@ceph.osd.3.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-3[115173]: 2023-07-14T19:20:10.246+0000 7f0d14620700 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:20:10.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T19:20:10.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T19:20:10.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:10.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:20:10.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:20:10.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:10.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:10.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:10.576 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:10.576 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.576 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:10.577 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:10.577 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:10.577 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.597 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T19:20:10.597 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T19:20:10.597 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:10.598 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:20:10.598 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:20:10.598 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.598 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.598 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:10.598 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:10.598 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.599 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:10.599 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:10.599 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:10.599 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:10.599 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:10.599 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:10.599 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:11.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:11 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi053 2023-07-14T19:20:11.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:11 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi053 to 8586M 2023-07-14T19:20:11.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:11 smithi104 ceph-mon[98435]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:20:11.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:11 smithi104 ceph-mon[98435]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T19:20:11.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:11.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:11.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:11 smithi104 ceph-mon[98435]: osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] boot 2023-07-14T19:20:11.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:11 smithi104 ceph-mon[98435]: osdmap e24: 4 total, 4 up, 4 in 2023-07-14T19:20:11.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:11.569 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.567+0000 7f81a3a37700 1 -- 172.21.15.70:0/225261904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c100240 msgr2=0x7f819c100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:11.569 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.567+0000 7f81a3a37700 1 --2- 172.21.15.70:0/225261904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c100240 0x7f819c100650 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8198009990 tx=0x7f8198009ca0 comp rx=0 tx=0).stop 2023-07-14T19:20:11.569 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.567+0000 7f81a3a37700 1 -- 172.21.15.70:0/225261904 shutdown_connections 2023-07-14T19:20:11.570 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.567+0000 7f81a3a37700 1 --2- 172.21.15.70:0/225261904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f819c0ff7c0 0x7f819c107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:11.570 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.567+0000 7f81a3a37700 1 --2- 172.21.15.70:0/225261904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c100240 0x7f819c100650 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:11.570 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.567+0000 7f81a3a37700 1 --2- 172.21.15.70:0/225261904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819c0fee10 0x7f819c0ff280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:11.570 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.567+0000 7f81a3a37700 1 -- 172.21.15.70:0/225261904 >> 172.21.15.70:0/225261904 conn(0x7f819c0f9990 msgr2=0x7f819c0fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:20:11.570 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.567+0000 7f81a3a37700 1 -- 172.21.15.70:0/225261904 shutdown_connections 2023-07-14T19:20:11.570 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.567+0000 7f81a3a37700 1 -- 172.21.15.70:0/225261904 wait complete. 2023-07-14T19:20:11.572 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a3a37700 1 Processor -- start 2023-07-14T19:20:11.572 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a3a37700 1 -- start start 2023-07-14T19:20:11.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a3a37700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f819c0fee10 0x7f819c1948c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:11.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a3a37700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c0ff7c0 0x7f819c194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:11.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a3a37700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819c100240 0x7f819c19be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:11.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a3a37700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c19c450 con 0x7f819c100240 2023-07-14T19:20:11.573 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a3a37700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c19c590 con 0x7f819c0ff7c0 2023-07-14T19:20:11.574 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a3a37700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c19c720 con 0x7f819c0fee10 2023-07-14T19:20:11.574 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a0fd2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c0ff7c0 0x7f819c194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:11.574 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a0fd2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c0ff7c0 0x7f819c194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56316/0 (socket says 172.21.15.70:56316) 2023-07-14T19:20:11.574 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a0fd2700 1 -- 172.21.15.70:0/2134885277 learned_addr learned my addr 172.21.15.70:0/2134885277 (peer_addr_for_me v2:172.21.15.70:0/0) 2023-07-14T19:20:11.574 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a17d3700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f819c0fee10 0x7f819c1948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:11.574 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a1fd4700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819c100240 0x7f819c19be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:11.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a0fd2700 1 -- 172.21.15.70:0/2134885277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f819c0fee10 msgr2=0x7f819c1948c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:11.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a0fd2700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f819c0fee10 0x7f819c1948c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:11.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a0fd2700 1 -- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819c100240 msgr2=0x7f819c19be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:11.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a0fd2700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819c100240 0x7f819c19be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:11.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a0fd2700 1 -- 172.21.15.70:0/2134885277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81980095f0 con 0x7f819c0ff7c0 2023-07-14T19:20:11.575 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a17d3700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f819c0fee10 0x7f819c1948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:20:11.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81a0fd2700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c0ff7c0 0x7f819c194e00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f818c00c920 tx=0x7f818c00cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:20:11.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.568+0000 7f81927fc700 1 -- 172.21.15.70:0/2134885277 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c007b00 con 0x7f819c0ff7c0 2023-07-14T19:20:11.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.569+0000 7f81927fc700 1 -- 172.21.15.70:0/2134885277 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f818c004cf0 con 0x7f819c0ff7c0 2023-07-14T19:20:11.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.569+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c19ca00 con 0x7f819c0ff7c0 2023-07-14T19:20:11.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.569+0000 7f81927fc700 1 -- 172.21.15.70:0/2134885277 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c0055e0 con 0x7f819c0ff7c0 2023-07-14T19:20:11.576 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.569+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c19cf50 con 0x7f819c0ff7c0 2023-07-14T19:20:11.577 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.569+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c10a380 con 0x7f819c0ff7c0 2023-07-14T19:20:11.577 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:11 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi053 2023-07-14T19:20:11.578 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:11 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi053 to 8586M 2023-07-14T19:20:11.578 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:11 smithi053 ceph-mon[96565]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:20:11.579 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:11 smithi053 ceph-mon[96565]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T19:20:11.579 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:11.579 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:11.579 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:11 smithi053 ceph-mon[96565]: osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] boot 2023-07-14T19:20:11.579 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:11 smithi053 ceph-mon[96565]: osdmap e24: 4 total, 4 up, 4 in 2023-07-14T19:20:11.580 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:11.581 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.571+0000 7f81927fc700 1 -- 172.21.15.70:0/2134885277 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f818c007c60 con 0x7f819c0ff7c0 2023-07-14T19:20:11.581 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.572+0000 7f81927fc700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f81880a11d0 0x7f81880a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:11.581 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.572+0000 7f81927fc700 1 -- 172.21.15.70:0/2134885277 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 3702+0+0 (secure 0 0 0) 0x7f818c0cb6b0 con 0x7f819c0ff7c0 2023-07-14T19:20:11.581 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.572+0000 7f81a17d3700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f81880a11d0 0x7f81880a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:11.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.572+0000 7f81a17d3700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f81880a11d0 0x7f81880a3680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f819800b5c0 tx=0x7f8198005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:20:11.582 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.575+0000 7f81927fc700 1 -- 172.21.15.70:0/2134885277 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f818c096080 con 0x7f819c0ff7c0 2023-07-14T19:20:11.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:11 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi053 2023-07-14T19:20:11.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:11 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi053 to 8586M 2023-07-14T19:20:11.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:11 smithi070 ceph-mon[98015]: from='osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:20:11.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:11 smithi070 ceph-mon[98015]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T19:20:11.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:11.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:11.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:11 smithi070 ceph-mon[98015]: osd.3 [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] boot 2023-07-14T19:20:11.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:11 smithi070 ceph-mon[98015]: osdmap e24: 4 total, 4 up, 4 in 2023-07-14T19:20:11.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:20:11.770 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:11.768+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f819c060a80 con 0x7f81880a11d0 2023-07-14T19:20:12.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:12 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:20:12.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:12 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:20:12.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:12 smithi104 ceph-mon[98435]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:12.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:12 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:20:12.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:12 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:20:12.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:12 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:12.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:12 smithi104 ceph-mon[98435]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T19:20:12.553 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:12 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:20:12.553 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:12 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:20:12.553 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:12 smithi070 ceph-mon[98015]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:12.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:12 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:20:12.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:12 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:20:12.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:12 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:12.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:12 smithi070 ceph-mon[98015]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T19:20:12.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:12 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:20:12.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:12 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:20:12.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:12 smithi053 ceph-mon[96565]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:12.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:12 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:20:12.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:12 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:20:12.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:12 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:12.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:12 smithi053 ceph-mon[96565]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T19:20:13.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:13 smithi104 ceph-mon[98435]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:20:13.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:13 smithi104 ceph-mon[98435]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T19:20:13.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:13 smithi053 ceph-mon[96565]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:20:13.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:13 smithi053 ceph-mon[96565]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T19:20:13.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:13 smithi070 ceph-mon[98015]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:20:13.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:13 smithi070 ceph-mon[98015]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T19:20:14.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:14 smithi104 ceph-mon[98435]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:14.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:14 smithi104 ceph-mon[98435]: from='client.? 172.21.15.70:0/3527123597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7dd3e40-bc97-4894-92ec-21c756a018ed"}]: dispatch 2023-07-14T19:20:14.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:14 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7dd3e40-bc97-4894-92ec-21c756a018ed"}]: dispatch 2023-07-14T19:20:14.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:14 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7dd3e40-bc97-4894-92ec-21c756a018ed"}]': finished 2023-07-14T19:20:14.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:14 smithi104 ceph-mon[98435]: osdmap e27: 5 total, 4 up, 5 in 2023-07-14T19:20:14.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:14 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:14.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:14 smithi053 ceph-mon[96565]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:14.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:14 smithi053 ceph-mon[96565]: from='client.? 172.21.15.70:0/3527123597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7dd3e40-bc97-4894-92ec-21c756a018ed"}]: dispatch 2023-07-14T19:20:14.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:14 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7dd3e40-bc97-4894-92ec-21c756a018ed"}]: dispatch 2023-07-14T19:20:14.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:14 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7dd3e40-bc97-4894-92ec-21c756a018ed"}]': finished 2023-07-14T19:20:14.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:14 smithi053 ceph-mon[96565]: osdmap e27: 5 total, 4 up, 5 in 2023-07-14T19:20:14.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:14.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:14 smithi070 ceph-mon[98015]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:14.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:14 smithi070 ceph-mon[98015]: from='client.? 172.21.15.70:0/3527123597' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7dd3e40-bc97-4894-92ec-21c756a018ed"}]: dispatch 2023-07-14T19:20:14.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:14 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7dd3e40-bc97-4894-92ec-21c756a018ed"}]: dispatch 2023-07-14T19:20:14.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:14 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7dd3e40-bc97-4894-92ec-21c756a018ed"}]': finished 2023-07-14T19:20:14.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:14 smithi070 ceph-mon[98015]: osdmap e27: 5 total, 4 up, 5 in 2023-07-14T19:20:14.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:14 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:15.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:15 smithi104 ceph-mon[98435]: from='client.? 172.21.15.70:0/215517456' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:20:15.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:15 smithi104 ceph-mon[98435]: osdmap e28: 5 total, 4 up, 5 in 2023-07-14T19:20:15.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:15 smithi053 ceph-mon[96565]: from='client.? 172.21.15.70:0/215517456' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:20:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:15 smithi053 ceph-mon[96565]: osdmap e28: 5 total, 4 up, 5 in 2023-07-14T19:20:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:15.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:15 smithi070 ceph-mon[98015]: from='client.? 172.21.15.70:0/215517456' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:20:15.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:15 smithi070 ceph-mon[98015]: osdmap e28: 5 total, 4 up, 5 in 2023-07-14T19:20:15.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:16.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:16 smithi053 ceph-mon[96565]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:20:16.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:16 smithi070 ceph-mon[98015]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:20:16.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:16 smithi104 ceph-mon[98435]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:20:18.477 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:18 smithi070 ceph-mon[98015]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2023-07-14T19:20:18.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:18 smithi104 ceph-mon[98435]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2023-07-14T19:20:18.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:18 smithi053 ceph-mon[96565]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2023-07-14T19:20:19.461 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:19 smithi070 ceph-mon[98015]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2023-07-14T19:20:19.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:19 smithi104 ceph-mon[98435]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2023-07-14T19:20:19.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:19 smithi053 ceph-mon[96565]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2023-07-14T19:20:20.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:20 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T19:20:20.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:20 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:20.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:20 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T19:20:20.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:20 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:21.010 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T19:20:21.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:20 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:21.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:21 smithi070 ceph-mon[98015]: Deploying daemon osd.4 on smithi070 2023-07-14T19:20:21.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:21 smithi070 ceph-mon[98015]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:20:21.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:21 smithi053 ceph-mon[96565]: Deploying daemon osd.4 on smithi070 2023-07-14T19:20:21.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:21 smithi053 ceph-mon[96565]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:20:22.010 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:21 smithi104 ceph-mon[98435]: Deploying daemon osd.4 on smithi070 2023-07-14T19:20:22.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:21 smithi104 ceph-mon[98435]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:20:23.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:23 smithi070 ceph-mon[98015]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:20:23.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:23 smithi104 ceph-mon[98435]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:20:23.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:23 smithi053 ceph-mon[96565]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:20:25.720 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:25.720 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:25.720 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:25 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:25.721 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:25 smithi070 ceph-mon[98015]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:26.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:26.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:26.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:25 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:26.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:25 smithi104 ceph-mon[98435]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:26.071 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:26.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:26.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:25 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:26.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:25 smithi053 ceph-mon[96565]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:27.383 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 4 on host 'smithi070' 2023-07-14T19:20:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.381+0000 7f81927fc700 1 -- 172.21.15.70:0/2134885277 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f819c060a80 con 0x7f81880a11d0 2023-07-14T19:20:27.385 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.383+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f81880a11d0 msgr2=0x7f81880a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:27.385 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.383+0000 7f81a3a37700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f81880a11d0 0x7f81880a3680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f819800b5c0 tx=0x7f8198005fb0 comp rx=0 tx=0).stop 2023-07-14T19:20:27.385 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.383+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c0ff7c0 msgr2=0x7f819c194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:27.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.383+0000 7f81a3a37700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c0ff7c0 0x7f819c194e00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f818c00c920 tx=0x7f818c00cce0 comp rx=0 tx=0).stop 2023-07-14T19:20:27.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.384+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 shutdown_connections 2023-07-14T19:20:27.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.384+0000 7f81a3a37700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f81880a11d0 0x7f81880a3680 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:27.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.384+0000 7f81a3a37700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819c100240 0x7f819c19be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:27.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.384+0000 7f81a3a37700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f819c0fee10 0x7f819c1948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:27.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.384+0000 7f81a3a37700 1 --2- 172.21.15.70:0/2134885277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c0ff7c0 0x7f819c194e00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:27.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.384+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 >> 172.21.15.70:0/2134885277 conn(0x7f819c0f9990 msgr2=0x7f819c0fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:20:27.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.384+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 shutdown_connections 2023-07-14T19:20:27.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:20:27.384+0000 7f81a3a37700 1 -- 172.21.15.70:0/2134885277 wait complete. 2023-07-14T19:20:27.778 DEBUG:teuthology.orchestra.run.smithi070:osd.4> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.4.service 2023-07-14T19:20:27.780 INFO:tasks.cephadm:Deploying osd.5 on smithi104 with /dev/vg_nvme/lv_3... 2023-07-14T19:20:27.781 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T19:20:27.830 INFO:journalctl@ceph.osd.4.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:20:28.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:28.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:28.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:27 smithi070 ceph-mon[98015]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:28.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:28.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:28.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:28.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:27 smithi104 ceph-mon[98435]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:28.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:28.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:28.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:28.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:27 smithi053 ceph-mon[96565]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:28.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:28.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:28.355 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:20:28.978 INFO:journalctl@ceph.osd.4.smithi070.stdout:Jul 14 19:20:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-4[108565]: 2023-07-14T19:20:28.622+0000 7f1d8aaab640 -1 osd.4 0 log_to_monitors true 2023-07-14T19:20:29.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:29 smithi070 ceph-mon[98015]: from='osd.4 [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:20:29.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:29 smithi070 ceph-mon[98015]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:20:29.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:29 smithi104 ceph-mon[98435]: from='osd.4 [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:20:29.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:29 smithi104 ceph-mon[98435]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:20:29.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:29 smithi053 ceph-mon[96565]: from='osd.4 [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:20:29.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:29 smithi053 ceph-mon[96565]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:20:29.865 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T19:20:29.865 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T19:20:29.865 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2023-07-14T19:20:29.865 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2023-07-14T19:20:29.865 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110556 s, 948 MB/s 2023-07-14T19:20:29.865 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2023-07-14T19:20:30.296 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2023-07-14T19:20:30.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T19:20:30.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T19:20:30.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:30.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='osd.4 [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:20:30.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:20:30.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:30.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:30.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:30.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:30.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:30.513 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T19:20:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T19:20:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='osd.4 [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:20:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:20:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:30.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:30.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.728 INFO:journalctl@ceph.osd.4.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-4[108565]: 2023-07-14T19:20:30.249+0000 7f1d7f922700 -1 osd.4 0 waiting for initial osdmap 2023-07-14T19:20:30.728 INFO:journalctl@ceph.osd.4.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-4[108565]: 2023-07-14T19:20:30.255+0000 7f1d78f13700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:20:30.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T19:20:30.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T19:20:30.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:30.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='osd.4 [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:20:30.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:20:30.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:20:30.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:30.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:30.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:30.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:30.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:30.843 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:20:31.373 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:31 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi070 2023-07-14T19:20:31.373 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:31 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi070 to 8584M 2023-07-14T19:20:31.373 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:31 smithi104 ceph-mon[98435]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:20:31.373 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:31 smithi104 ceph-mon[98435]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T19:20:31.374 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:31 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:31.374 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:31 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:31 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi070 2023-07-14T19:20:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:31 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi070 to 8584M 2023-07-14T19:20:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:31 smithi053 ceph-mon[96565]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:20:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:31 smithi053 ceph-mon[96565]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T19:20:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:31 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:31.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:31 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:31.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:31 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi070 2023-07-14T19:20:31.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:31 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi070 to 8584M 2023-07-14T19:20:31.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:31 smithi070 ceph-mon[98015]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:20:31.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:31 smithi070 ceph-mon[98015]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T19:20:31.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:31 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:31.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:31 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:32.126 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.124+0000 7fc309e76700 1 -- 172.21.15.104:0/1191354058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3040ff7c0 msgr2=0x7fc304107070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:32.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.124+0000 7fc309e76700 1 --2- 172.21.15.104:0/1191354058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3040ff7c0 0x7fc304107070 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00b0a0 tx=0x7fc2fc00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:20:32.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 -- 172.21.15.104:0/1191354058 shutdown_connections 2023-07-14T19:20:32.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 --2- 172.21.15.104:0/1191354058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc3040ff7c0 0x7fc304107070 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:32.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 --2- 172.21.15.104:0/1191354058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc304100240 0x7fc304100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:32.127 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 --2- 172.21.15.104:0/1191354058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3040fee10 0x7fc3040ff280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:32.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 -- 172.21.15.104:0/1191354058 >> 172.21.15.104:0/1191354058 conn(0x7fc3040f9990 msgr2=0x7fc3040fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:20:32.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 -- 172.21.15.104:0/1191354058 shutdown_connections 2023-07-14T19:20:32.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 -- 172.21.15.104:0/1191354058 wait complete. 2023-07-14T19:20:32.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 Processor -- start 2023-07-14T19:20:32.128 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 -- start start 2023-07-14T19:20:32.129 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3040fee10 0x7fc304194980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:32.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.125+0000 7fc309e76700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3040ff7c0 0x7fc304194ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:32.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc309e76700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304100240 0x7fc30419bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:32.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc309e76700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30419c510 con 0x7fc3040fee10 2023-07-14T19:20:32.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc309e76700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30419c650 con 0x7fc3040ff7c0 2023-07-14T19:20:32.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc309e76700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30419c7e0 con 0x7fc304100240 2023-07-14T19:20:32.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc303fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304100240 0x7fc30419bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:32.130 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc303fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304100240 0x7fc30419bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:56990/0 (socket says 172.21.15.104:56990) 2023-07-14T19:20:32.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc303fff700 1 -- 172.21.15.104:0/2716514829 learned_addr learned my addr 172.21.15.104:0/2716514829 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:20:32.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc3037fe700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3040fee10 0x7fc304194980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:32.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc302ffd700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3040ff7c0 0x7fc304194ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:32.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc303fff700 1 -- 172.21.15.104:0/2716514829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3040ff7c0 msgr2=0x7fc304194ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:32.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc303fff700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3040ff7c0 0x7fc304194ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:32.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc303fff700 1 -- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3040fee10 msgr2=0x7fc304194980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:32.131 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc303fff700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3040fee10 0x7fc304194980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:32.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc303fff700 1 -- 172.21.15.104:0/2716514829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc009d00 con 0x7fc304100240 2023-07-14T19:20:32.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc303fff700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304100240 0x7fc30419bf40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc007ab0 tx=0x7fc2fc004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:20:32.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc300ff9700 1 -- 172.21.15.104:0/2716514829 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc00e040 con 0x7fc304100240 2023-07-14T19:20:32.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc300ff9700 1 -- 172.21.15.104:0/2716514829 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2fc00cec0 con 0x7fc304100240 2023-07-14T19:20:32.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc300ff9700 1 -- 172.21.15.104:0/2716514829 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc01bb30 con 0x7fc304100240 2023-07-14T19:20:32.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3040fde40 con 0x7fc304100240 2023-07-14T19:20:32.132 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.126+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3040fe2f0 con 0x7fc304100240 2023-07-14T19:20:32.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.127+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30410a380 con 0x7fc304100240 2023-07-14T19:20:32.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.128+0000 7fc300ff9700 1 -- 172.21.15.104:0/2716514829 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc2fc01bc90 con 0x7fc304100240 2023-07-14T19:20:32.133 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.129+0000 7fc300ff9700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc2ec0a11d0 0x7fc2ec0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:32.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.129+0000 7fc300ff9700 1 -- 172.21.15.104:0/2716514829 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fc2fc0cd3a0 con 0x7fc304100240 2023-07-14T19:20:32.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.129+0000 7fc3037fe700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc2ec0a11d0 0x7fc2ec0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:32.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.129+0000 7fc3037fe700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc2ec0a11d0 0x7fc2ec0a3680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005950 tx=0x7fc2f800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:20:32.134 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.133+0000 7fc300ff9700 1 -- 172.21.15.104:0/2716514829 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc2fc097bf0 con 0x7fc304100240 2023-07-14T19:20:32.334 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:32.332+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc304060a80 con 0x7fc2ec0a11d0 2023-07-14T19:20:32.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:32 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:20:32.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:32 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:20:32.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:32 smithi104 ceph-mon[98435]: osd.4 [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] boot 2023-07-14T19:20:32.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:32 smithi104 ceph-mon[98435]: osdmap e31: 5 total, 5 up, 5 in 2023-07-14T19:20:32.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:32.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:32 smithi104 ceph-mon[98435]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:32 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:20:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:32 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:20:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:32 smithi053 ceph-mon[96565]: osd.4 [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] boot 2023-07-14T19:20:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:32 smithi053 ceph-mon[96565]: osdmap e31: 5 total, 5 up, 5 in 2023-07-14T19:20:32.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:32.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:32 smithi053 ceph-mon[96565]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:32 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:20:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:32 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:20:32.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:32 smithi070 ceph-mon[98015]: osd.4 [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] boot 2023-07-14T19:20:32.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:32 smithi070 ceph-mon[98015]: osdmap e31: 5 total, 5 up, 5 in 2023-07-14T19:20:32.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:20:32.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:32 smithi070 ceph-mon[98015]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:33.456 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:33 smithi104 ceph-mon[98435]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T19:20:33.457 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:20:33.457 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:20:33.457 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:33 smithi053 ceph-mon[96565]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T19:20:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:20:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:20:33.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:33 smithi070 ceph-mon[98015]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T19:20:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:33 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:20:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:33 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:20:33.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:33 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:34.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:34 smithi053 ceph-mon[96565]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:20:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:34 smithi053 ceph-mon[96565]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:34.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:34 smithi070 ceph-mon[98015]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:20:34.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:34 smithi070 ceph-mon[98015]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:34.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:34 smithi104 ceph-mon[98435]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:20:34.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:34 smithi104 ceph-mon[98435]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:35 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/3779625006' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "530ff111-a77f-4eee-894f-0098aa1febd6"}]: dispatch 2023-07-14T19:20:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:35 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "530ff111-a77f-4eee-894f-0098aa1febd6"}]: dispatch 2023-07-14T19:20:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:35 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "530ff111-a77f-4eee-894f-0098aa1febd6"}]': finished 2023-07-14T19:20:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:35 smithi053 ceph-mon[96565]: osdmap e33: 6 total, 5 up, 6 in 2023-07-14T19:20:35.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:35 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/3779625006' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "530ff111-a77f-4eee-894f-0098aa1febd6"}]: dispatch 2023-07-14T19:20:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:35 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "530ff111-a77f-4eee-894f-0098aa1febd6"}]: dispatch 2023-07-14T19:20:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:35 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "530ff111-a77f-4eee-894f-0098aa1febd6"}]': finished 2023-07-14T19:20:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:35 smithi070 ceph-mon[98015]: osdmap e33: 6 total, 5 up, 6 in 2023-07-14T19:20:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:35.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:35 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/3779625006' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "530ff111-a77f-4eee-894f-0098aa1febd6"}]: dispatch 2023-07-14T19:20:35.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:35 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "530ff111-a77f-4eee-894f-0098aa1febd6"}]: dispatch 2023-07-14T19:20:35.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:35 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "530ff111-a77f-4eee-894f-0098aa1febd6"}]': finished 2023-07-14T19:20:35.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:35 smithi104 ceph-mon[98435]: osdmap e33: 6 total, 5 up, 6 in 2023-07-14T19:20:35.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:36 smithi053 ceph-mon[96565]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:36 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/2398123471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:20:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:36 smithi070 ceph-mon[98015]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:36 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/2398123471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:20:36.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:36 smithi104 ceph-mon[98435]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:36.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:36 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/2398123471' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:20:38.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:38 smithi104 ceph-mon[98435]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:38 smithi053 ceph-mon[96565]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:38.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:38 smithi070 ceph-mon[98015]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:39.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:39 smithi070 ceph-mon[98015]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:39.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:39 smithi104 ceph-mon[98435]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:39.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:39 smithi053 ceph-mon[96565]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:41 smithi070 ceph-mon[98015]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T19:20:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:41 smithi104 ceph-mon[98435]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T19:20:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:41.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:41 smithi053 ceph-mon[96565]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:41.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T19:20:41.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:43.460 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:43 smithi104 ceph-mon[98435]: Deploying daemon osd.5 on smithi104 2023-07-14T19:20:43.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:43 smithi053 ceph-mon[96565]: Deploying daemon osd.5 on smithi104 2023-07-14T19:20:43.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:43 smithi070 ceph-mon[98015]: Deploying daemon osd.5 on smithi104 2023-07-14T19:20:44.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:44 smithi104 ceph-mon[98435]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:44.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:44 smithi053 ceph-mon[96565]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:44.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:44 smithi070 ceph-mon[98015]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:45.661 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:45 smithi104 ceph-mon[98435]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:45.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:45 smithi070 ceph-mon[98015]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:45.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:45 smithi053 ceph-mon[96565]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:46.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:46.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:46.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:46 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:46.979 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:46.979 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:46.980 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:46 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:47.071 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:47.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:47.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:46 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:48.346 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.345+0000 7fc300ff9700 1 -- 172.21.15.104:0/2716514829 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc304060a80 con 0x7fc2ec0a11d0 2023-07-14T19:20:48.346 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 5 on host 'smithi104' 2023-07-14T19:20:48.348 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc2ec0a11d0 msgr2=0x7fc2ec0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:48.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc2ec0a11d0 0x7fc2ec0a3680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8005950 tx=0x7fc2f800b410 comp rx=0 tx=0).stop 2023-07-14T19:20:48.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304100240 msgr2=0x7fc30419bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:48.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304100240 0x7fc30419bf40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc007ab0 tx=0x7fc2fc004d00 comp rx=0 tx=0).stop 2023-07-14T19:20:48.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 shutdown_connections 2023-07-14T19:20:48.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc2ec0a11d0 0x7fc2ec0a3680 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:48.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc304100240 0x7fc30419bf40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:48.349 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3040fee10 0x7fc304194980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:48.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 --2- 172.21.15.104:0/2716514829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc3040ff7c0 0x7fc304194ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:48.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 >> 172.21.15.104:0/2716514829 conn(0x7fc3040f9990 msgr2=0x7fc3040fbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:20:48.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 shutdown_connections 2023-07-14T19:20:48.350 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:20:48.347+0000 7fc309e76700 1 -- 172.21.15.104:0/2716514829 wait complete. 2023-07-14T19:20:48.503 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:48 smithi104 ceph-mon[98435]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:48.504 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:48.504 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:48.505 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:48.505 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:48.505 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:48.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:48 smithi053 ceph-mon[96565]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:48.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:48.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:48.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:48.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:48.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:48.722 DEBUG:teuthology.orchestra.run.smithi104:osd.5> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.5.service 2023-07-14T19:20:48.726 INFO:tasks.cephadm:Deploying osd.6 on smithi053 with /dev/vg_nvme/lv_2... 2023-07-14T19:20:48.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:20:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:48 smithi070 ceph-mon[98015]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:48.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:48.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:48.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:48.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:48.754 INFO:journalctl@ceph.osd.5.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:20:49.318 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:20:49.486 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:49.486 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:49.486 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:49 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:49.486 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:49 smithi104 ceph-mon[98435]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:49.694 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:49.694 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:49.695 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:49 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:49.695 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:49 smithi053 ceph-mon[96565]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:49.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:49.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:49.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:49 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:20:49.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:49 smithi070 ceph-mon[98015]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:49.760 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:20:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:20:49.716+0000 7f6bd15e2640 -1 osd.5 0 log_to_monitors true 2023-07-14T19:20:50.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:50 smithi053 ceph-mon[96565]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:20:50.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:50 smithi070 ceph-mon[98015]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:20:50.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:50 smithi104 ceph-mon[98435]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:20:50.827 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:20:50.828 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:20:50.828 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2023-07-14T19:20:50.828 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2023-07-14T19:20:50.828 INFO:teuthology.orchestra.run.smithi053.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00838023 s, 1.3 GB/s 2023-07-14T19:20:50.829 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2023-07-14T19:20:51.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi053:vg_nvme/lv_2 2023-07-14T19:20:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T19:20:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T19:20:51.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:51.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:20:51.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi104 2023-07-14T19:20:51.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:51.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:51.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi104 to 8584M 2023-07-14T19:20:51.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:51.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:51.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:51 smithi053 ceph-mon[96565]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:51.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T19:20:51.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T19:20:51.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:51.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:20:51.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi104 2023-07-14T19:20:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi104 to 8584M 2023-07-14T19:20:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:51.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:51.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:51 smithi070 ceph-mon[98015]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:51.760 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:20:51.354+0000 7f6bc6459700 -1 osd.5 0 waiting for initial osdmap 2023-07-14T19:20:51.761 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:20:51.365+0000 7f6bbf249700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:20:51.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T19:20:51.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T19:20:51.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:51.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:20:51.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi104 2023-07-14T19:20:51.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:51.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:20:51.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi104 to 8584M 2023-07-14T19:20:51.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:51.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:20:51.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:20:51.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:51 smithi104 ceph-mon[98435]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:20:51.839 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:20:52.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:52 smithi053 ceph-mon[96565]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:20:52.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:52 smithi053 ceph-mon[96565]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T19:20:52.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:52.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:52.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:52 smithi053 ceph-mon[96565]: osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] boot 2023-07-14T19:20:52.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:52 smithi053 ceph-mon[96565]: osdmap e36: 6 total, 6 up, 6 in 2023-07-14T19:20:52.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:52.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:52 smithi070 ceph-mon[98015]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:20:52.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:52 smithi070 ceph-mon[98015]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T19:20:52.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:52.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:52.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:52 smithi070 ceph-mon[98015]: osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] boot 2023-07-14T19:20:52.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:52 smithi070 ceph-mon[98015]: osdmap e36: 6 total, 6 up, 6 in 2023-07-14T19:20:52.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:52.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:52 smithi104 ceph-mon[98435]: from='osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:20:52.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:52 smithi104 ceph-mon[98435]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T19:20:52.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:52.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:52.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:52 smithi104 ceph-mon[98435]: osd.5 [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] boot 2023-07-14T19:20:52.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:52 smithi104 ceph-mon[98435]: osdmap e36: 6 total, 6 up, 6 in 2023-07-14T19:20:52.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:20:53.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 -- 172.21.15.53:0/1700968161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c105980 msgr2=0x7fd04c107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:53.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 --2- 172.21.15.53:0/1700968161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c105980 0x7fd04c107de0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd04400b0a0 tx=0x7fd04400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:20:53.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 -- 172.21.15.53:0/1700968161 shutdown_connections 2023-07-14T19:20:53.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 --2- 172.21.15.53:0/1700968161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c105980 0x7fd04c107de0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:53.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 --2- 172.21.15.53:0/1700968161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c100740 0x7fd04c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:53.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 --2- 172.21.15.53:0/1700968161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd04c103060 0x7fd04c105440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:53.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 -- 172.21.15.53:0/1700968161 >> 172.21.15.53:0/1700968161 conn(0x7fd04c0fa1e0 msgr2=0x7fd04c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:20:53.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 -- 172.21.15.53:0/1700968161 shutdown_connections 2023-07-14T19:20:53.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 -- 172.21.15.53:0/1700968161 wait complete. 2023-07-14T19:20:53.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.049+0000 7fd051007700 1 Processor -- start 2023-07-14T19:20:53.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd051007700 1 -- start start 2023-07-14T19:20:53.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd051007700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd04c100740 0x7fd04c1000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:53.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd051007700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c103060 0x7fd04c0fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:53.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd051007700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c105980 0x7fd04c0fec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:53.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd051007700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0ff3b0 con 0x7fd04c103060 2023-07-14T19:20:53.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd051007700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0ff520 con 0x7fd04c105980 2023-07-14T19:20:53.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd051007700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c19fa70 con 0x7fd04c100740 2023-07-14T19:20:53.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c103060 0x7fd04c0fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:53.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04a59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c103060 0x7fd04c0fe6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49508/0 (socket says 172.21.15.53:49508) 2023-07-14T19:20:53.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04a59c700 1 -- 172.21.15.53:0/3918486591 learned_addr learned my addr 172.21.15.53:0/3918486591 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:20:53.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04b59e700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c105980 0x7fd04c0fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:53.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04ad9d700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd04c100740 0x7fd04c1000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:53.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04a59c700 1 -- 172.21.15.53:0/3918486591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd04c100740 msgr2=0x7fd04c1000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:53.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04a59c700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd04c100740 0x7fd04c1000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:53.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04a59c700 1 -- 172.21.15.53:0/3918486591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c105980 msgr2=0x7fd04c0fec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:20:53.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04a59c700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c105980 0x7fd04c0fec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:20:53.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04a59c700 1 -- 172.21.15.53:0/3918486591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd044009d00 con 0x7fd04c103060 2023-07-14T19:20:53.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.050+0000 7fd04a59c700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c103060 0x7fd04c0fe6f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c9b0 tx=0x7fd04000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:20:53.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.051+0000 7fd033fff700 1 -- 172.21.15.53:0/3918486591 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040007b00 con 0x7fd04c103060 2023-07-14T19:20:53.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.051+0000 7fd033fff700 1 -- 172.21.15.53:0/3918486591 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd040007c60 con 0x7fd04c103060 2023-07-14T19:20:53.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.051+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c19fd50 con 0x7fd04c103060 2023-07-14T19:20:53.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.051+0000 7fd033fff700 1 -- 172.21.15.53:0/3918486591 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040018660 con 0x7fd04c103060 2023-07-14T19:20:53.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.051+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c1a01f0 con 0x7fd04c103060 2023-07-14T19:20:53.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.052+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c0fbd50 con 0x7fd04c103060 2023-07-14T19:20:53.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.053+0000 7fd033fff700 1 -- 172.21.15.53:0/3918486591 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd04000f450 con 0x7fd04c103060 2023-07-14T19:20:53.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.053+0000 7fd033fff700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd0340a1150 0x7fd0340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:20:53.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.053+0000 7fd04ad9d700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd0340a1150 0x7fd0340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:20:53.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.053+0000 7fd033fff700 1 -- 172.21.15.53:0/3918486591 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4566+0+0 (secure 0 0 0) 0x7fd0400cbc90 con 0x7fd04c103060 2023-07-14T19:20:53.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.054+0000 7fd04ad9d700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd0340a1150 0x7fd0340a3600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd03c005950 tx=0x7fd03c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:20:53.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.057+0000 7fd033fff700 1 -- 172.21.15.53:0/3918486591 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd040096310 con 0x7fd04c103060 2023-07-14T19:20:53.265 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:20:53.264+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd04c060a80 con 0x7fd0340a1150 2023-07-14T19:20:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:53 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:20:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:53 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:20:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:53 smithi070 ceph-mon[98015]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T19:20:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:53 smithi070 ceph-mon[98015]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:20:53.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:20:53.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:20:53.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:53.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:53 smithi070 ceph-mon[98015]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:20:53.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:53 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:20:53.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:53 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:20:53.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:53 smithi104 ceph-mon[98435]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T19:20:53.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:53 smithi104 ceph-mon[98435]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:20:53.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:20:53.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:20:53.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:53.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:53 smithi104 ceph-mon[98435]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:20:53.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:53 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:20:53.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:53 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:20:53.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:53 smithi053 ceph-mon[96565]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T19:20:53.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:53 smithi053 ceph-mon[96565]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:20:53.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:20:53.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:20:53.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:20:53.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:53 smithi053 ceph-mon[96565]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:20:55.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:55 smithi104 ceph-mon[98435]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T19:20:55.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:55 smithi053 ceph-mon[96565]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T19:20:55.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:55 smithi070 ceph-mon[98015]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T19:20:56.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:56 smithi104 ceph-mon[98435]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T19:20:56.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:56 smithi104 ceph-mon[98435]: pgmap v126: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:20:56.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:56 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2465562408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7ab09b5-1dfc-4d44-a248-7aac58e27749"}]: dispatch 2023-07-14T19:20:56.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:56 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2465562408' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7ab09b5-1dfc-4d44-a248-7aac58e27749"}]': finished 2023-07-14T19:20:56.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:56 smithi104 ceph-mon[98435]: osdmap e40: 7 total, 6 up, 7 in 2023-07-14T19:20:56.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:56 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:20:56.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:56 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/456649687' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:20:56.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:56 smithi053 ceph-mon[96565]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T19:20:56.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:56 smithi053 ceph-mon[96565]: pgmap v126: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:20:56.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:56 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2465562408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7ab09b5-1dfc-4d44-a248-7aac58e27749"}]: dispatch 2023-07-14T19:20:56.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:56 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2465562408' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7ab09b5-1dfc-4d44-a248-7aac58e27749"}]': finished 2023-07-14T19:20:56.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:56 smithi053 ceph-mon[96565]: osdmap e40: 7 total, 6 up, 7 in 2023-07-14T19:20:56.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:56 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:20:56.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:56 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/456649687' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:20:56.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:56 smithi070 ceph-mon[98015]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T19:20:56.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:56 smithi070 ceph-mon[98015]: pgmap v126: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:20:56.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:56 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2465562408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7ab09b5-1dfc-4d44-a248-7aac58e27749"}]: dispatch 2023-07-14T19:20:56.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:56 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2465562408' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7ab09b5-1dfc-4d44-a248-7aac58e27749"}]': finished 2023-07-14T19:20:56.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:56 smithi070 ceph-mon[98015]: osdmap e40: 7 total, 6 up, 7 in 2023-07-14T19:20:56.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:56 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:20:56.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:56 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/456649687' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:20:58.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:58 smithi104 ceph-mon[98435]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:20:58.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:58 smithi053 ceph-mon[96565]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:20:58.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:58 smithi070 ceph-mon[98015]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:20:59.510 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:20:59 smithi053 ceph-mon[96565]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:20:59.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:20:59 smithi070 ceph-mon[98015]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:20:59.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:20:59 smithi104 ceph-mon[98435]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:21:01.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:01 smithi070 ceph-mon[98015]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-14T19:21:01.757 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:01 smithi053 ceph-mon[96565]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-14T19:21:01.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:01 smithi104 ceph-mon[98435]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-14T19:21:02.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:02 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T19:21:02.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:02 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:02.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:02 smithi053 ceph-mon[96565]: Deploying daemon osd.6 on smithi053 2023-07-14T19:21:02.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:02 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T19:21:02.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:02 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:02.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:02 smithi070 ceph-mon[98015]: Deploying daemon osd.6 on smithi053 2023-07-14T19:21:02.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:02 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T19:21:02.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:02 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:02.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:02 smithi104 ceph-mon[98435]: Deploying daemon osd.6 on smithi053 2023-07-14T19:21:03.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:03 smithi053 ceph-mon[96565]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:21:03.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:03 smithi070 ceph-mon[98015]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:21:03.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:03 smithi104 ceph-mon[98435]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:21:05.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:05 smithi053 ceph-mon[96565]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:21:05.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:05 smithi070 ceph-mon[98015]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:21:05.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:05 smithi104 ceph-mon[98435]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:21:07.315 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:07.315 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:07.316 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:07 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:07.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:07 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:07.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:07 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:07.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:07 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:07.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:07 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:07.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:07 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:07.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:07 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:08.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:08 smithi104 ceph-mon[98435]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:21:08.521 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:08 smithi053 ceph-mon[96565]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:21:08.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:08 smithi070 ceph-mon[98015]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:21:08.968 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.966+0000 7fd033fff700 1 -- 172.21.15.53:0/3918486591 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd04c060a80 con 0x7fd0340a1150 2023-07-14T19:21:08.969 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 6 on host 'smithi053' 2023-07-14T19:21:08.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd0340a1150 msgr2=0x7fd0340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:08.971 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd0340a1150 0x7fd0340a3600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd03c005950 tx=0x7fd03c0058e0 comp rx=0 tx=0).stop 2023-07-14T19:21:08.971 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c103060 msgr2=0x7fd04c0fe6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:08.971 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c103060 0x7fd04c0fe6f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c9b0 tx=0x7fd04000ccc0 comp rx=0 tx=0).stop 2023-07-14T19:21:08.971 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 shutdown_connections 2023-07-14T19:21:08.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd0340a1150 0x7fd0340a3600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:08.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd04c105980 0x7fd04c0fec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:08.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd04c100740 0x7fd04c1000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:08.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 --2- 172.21.15.53:0/3918486591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd04c103060 0x7fd04c0fe6f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 >> 172.21.15.53:0/3918486591 conn(0x7fd04c0fa1e0 msgr2=0x7fd04c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:21:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 shutdown_connections 2023-07-14T19:21:08.974 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:08.969+0000 7fd051007700 1 -- 172.21.15.53:0/3918486591 wait complete. 2023-07-14T19:21:09.444 DEBUG:teuthology.orchestra.run.smithi053:osd.6> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.6.service 2023-07-14T19:21:09.464 INFO:tasks.cephadm:Deploying osd.7 on smithi070 with /dev/vg_nvme/lv_2... 2023-07-14T19:21:09.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:21:09.496 INFO:journalctl@ceph.osd.6.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:21:09.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:09.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:09.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:09.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:09.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:09 smithi053 ceph-mon[96565]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:21:09.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:09 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:09.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:09 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:09.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:09 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:09.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:09 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:09.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:09 smithi070 ceph-mon[98015]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:21:10.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:09 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:10.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:09 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:10.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:09 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:10.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:09 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:10.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:09 smithi104 ceph-mon[98435]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:21:10.023 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:21:10.206 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:21:10 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:21:10.149+0000 7f00c7ac0640 -1 osd.6 0 log_to_monitors true 2023-07-14T19:21:11.209 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:10 smithi053 ceph-mon[96565]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:21:11.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:10 smithi070 ceph-mon[98015]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:21:11.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:10 smithi104 ceph-mon[98435]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:21:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:21:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:21:11.596 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2023-07-14T19:21:11.596 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2023-07-14T19:21:11.597 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00817569 s, 1.3 GB/s 2023-07-14T19:21:11.597 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2023-07-14T19:21:11.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi070:vg_nvme/lv_2 2023-07-14T19:21:12.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:21:12.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: osdmap e41: 7 total, 6 up, 7 in 2023-07-14T19:21:12.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:12.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:21:12.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:21:12.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:12.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:12.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:12.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:12.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:12.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:12.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:12.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:12.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:21:12.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: osdmap e41: 7 total, 6 up, 7 in 2023-07-14T19:21:12.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:12.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:21:12.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:21:12.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:12.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:12.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:12.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:12.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:12.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:12.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:12.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:12.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.322 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:21:11.966+0000 7f00bc937700 -1 osd.6 0 waiting for initial osdmap 2023-07-14T19:21:12.322 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:21:11.975+0000 7f00b872d700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:21:12.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:21:12.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: osdmap e41: 7 total, 6 up, 7 in 2023-07-14T19:21:12.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:12.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:21:12.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:21:12.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:12.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:12.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:12.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:12.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:12.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:12.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:12.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:12.325 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:12.529 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:21:13.025 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:12 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi053 2023-07-14T19:21:13.026 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:12 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi053 to 5724M 2023-07-14T19:21:13.026 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:12 smithi070 ceph-mon[98015]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:21:13.026 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:12 smithi070 ceph-mon[98015]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T19:21:13.026 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:12 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:13.026 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:12 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:13.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:12 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi053 2023-07-14T19:21:13.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:12 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi053 to 5724M 2023-07-14T19:21:13.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:12 smithi104 ceph-mon[98435]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:21:13.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:12 smithi104 ceph-mon[98435]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T19:21:13.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:12 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:13.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:12 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:13.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:12 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi053 2023-07-14T19:21:13.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:12 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi053 to 5724M 2023-07-14T19:21:13.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:12 smithi053 ceph-mon[96565]: from='osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:21:13.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:12 smithi053 ceph-mon[96565]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T19:21:13.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:12 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:13.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:12 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:13.733 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.731+0000 7f6efd2fb700 1 -- 172.21.15.70:0/542206971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80ff800 msgr2=0x7f6ef81070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:13.734 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.731+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/542206971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80ff800 0x7f6ef81070b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000b0a0 tx=0x7f6ef000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:21:13.734 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.731+0000 7f6efd2fb700 1 -- 172.21.15.70:0/542206971 shutdown_connections 2023-07-14T19:21:13.734 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.731+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/542206971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80ff800 0x7f6ef81070b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:13.735 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.731+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/542206971 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef8100280 0x7f6ef8100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:13.735 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.731+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/542206971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ef80fee50 0x7f6ef80ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:13.735 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.731+0000 7f6efd2fb700 1 -- 172.21.15.70:0/542206971 >> 172.21.15.70:0/542206971 conn(0x7f6ef80f99b0 msgr2=0x7f6ef80fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:21:13.735 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.731+0000 7f6efd2fb700 1 -- 172.21.15.70:0/542206971 shutdown_connections 2023-07-14T19:21:13.736 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.731+0000 7f6efd2fb700 1 -- 172.21.15.70:0/542206971 wait complete. 2023-07-14T19:21:13.736 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6efd2fb700 1 Processor -- start 2023-07-14T19:21:13.736 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6efd2fb700 1 -- start start 2023-07-14T19:21:13.739 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6efd2fb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80fee50 0x7f6ef80fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:13.739 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6efd2fb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ef80ff800 0x7f6ef819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:13.739 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6efd2fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef8100280 0x7f6ef819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6efd2fb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef819d5f0 con 0x7f6ef80ff800 2023-07-14T19:21:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6efd2fb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef819d760 con 0x7f6ef80fee50 2023-07-14T19:21:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6efd2fb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef819d920 con 0x7f6ef8100280 2023-07-14T19:21:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6ef6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80fee50 0x7f6ef80fe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6ef6ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80fee50 0x7f6ef80fe8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56416/0 (socket says 172.21.15.70:56416) 2023-07-14T19:21:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.732+0000 7f6ef6ffd700 1 -- 172.21.15.70:0/945694948 learned_addr learned my addr 172.21.15.70:0/945694948 (peer_addr_for_me v2:172.21.15.70:0/0) 2023-07-14T19:21:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6ef67fc700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ef80ff800 0x7f6ef819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6ef77fe700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef8100280 0x7f6ef819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6ef6ffd700 1 -- 172.21.15.70:0/945694948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef8100280 msgr2=0x7f6ef819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6ef6ffd700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef8100280 0x7f6ef819ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6ef6ffd700 1 -- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ef80ff800 msgr2=0x7f6ef819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6ef6ffd700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ef80ff800 0x7f6ef819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6ef6ffd700 1 -- 172.21.15.70:0/945694948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef0009d00 con 0x7f6ef80fee50 2023-07-14T19:21:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6ef6ffd700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80fee50 0x7f6ef80fe8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800d980 tx=0x7f6ee800dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:21:13.742 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6eeffff700 1 -- 172.21.15.70:0/945694948 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8009b40 con 0x7f6ef80fee50 2023-07-14T19:21:13.742 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6eeffff700 1 -- 172.21.15.70:0/945694948 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ee800de60 con 0x7f6ef80fee50 2023-07-14T19:21:13.742 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef819dc00 con 0x7f6ef80fee50 2023-07-14T19:21:13.742 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6eeffff700 1 -- 172.21.15.70:0/945694948 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8005580 con 0x7f6ef80fee50 2023-07-14T19:21:13.742 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.733+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef819e0b0 con 0x7f6ef80fee50 2023-07-14T19:21:13.742 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.734+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef8106170 con 0x7f6ef80fee50 2023-07-14T19:21:13.743 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.735+0000 7f6eeffff700 1 -- 172.21.15.70:0/945694948 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6ee8004750 con 0x7f6ef80fee50 2023-07-14T19:21:13.743 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.736+0000 7f6eeffff700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6ee40a1150 0x7f6ee40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:13.743 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.736+0000 7f6eeffff700 1 -- 172.21.15.70:0/945694948 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f6ee80cc090 con 0x7f6ef80fee50 2023-07-14T19:21:13.743 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.736+0000 7f6ef67fc700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6ee40a1150 0x7f6ee40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:13.743 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.737+0000 7f6ef67fc700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6ee40a1150 0x7f6ee40a3600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0005950 tx=0x7f6ee00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:21:13.744 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.739+0000 7f6eeffff700 1 -- 172.21.15.70:0/945694948 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6ee8096450 con 0x7f6ef80fee50 2023-07-14T19:21:13.934 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:13.932+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ef8060a80 con 0x7f6ee40a1150 2023-07-14T19:21:14.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:13 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:21:14.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:13 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:21:14.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:13 smithi070 ceph-mon[98015]: osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] boot 2023-07-14T19:21:14.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:13 smithi070 ceph-mon[98015]: osdmap e43: 7 total, 7 up, 7 in 2023-07-14T19:21:14.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:14.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:13 smithi070 ceph-mon[98015]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:21:14.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:21:14.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:21:14.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:13 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:21:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:13 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:21:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:13 smithi104 ceph-mon[98435]: osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] boot 2023-07-14T19:21:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:13 smithi104 ceph-mon[98435]: osdmap e43: 7 total, 7 up, 7 in 2023-07-14T19:21:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:13 smithi104 ceph-mon[98435]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:21:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:21:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:21:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:14.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:13 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:21:14.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:13 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:21:14.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:13 smithi053 ceph-mon[96565]: osd.6 [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] boot 2023-07-14T19:21:14.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:13 smithi053 ceph-mon[96565]: osdmap e43: 7 total, 7 up, 7 in 2023-07-14T19:21:14.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:21:14.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:13 smithi053 ceph-mon[96565]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:21:14.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:21:14.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:21:14.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:15.227 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:14 smithi070 ceph-mon[98015]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:21:15.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:14 smithi070 ceph-mon[98015]: osdmap e44: 7 total, 7 up, 7 in 2023-07-14T19:21:15.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:14 smithi104 ceph-mon[98435]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:21:15.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:14 smithi104 ceph-mon[98435]: osdmap e44: 7 total, 7 up, 7 in 2023-07-14T19:21:15.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:14 smithi053 ceph-mon[96565]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:21:15.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:14 smithi053 ceph-mon[96565]: osdmap e44: 7 total, 7 up, 7 in 2023-07-14T19:21:16.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:15 smithi104 ceph-mon[98435]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:16.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:15 smithi053 ceph-mon[96565]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:16.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:15 smithi070 ceph-mon[98015]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:17.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:16 smithi104 ceph-mon[98435]: from='client.? 172.21.15.70:0/4212596606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b18c85a-6628-4f12-b07f-d292fcd3a05b"}]: dispatch 2023-07-14T19:21:17.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:16 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b18c85a-6628-4f12-b07f-d292fcd3a05b"}]: dispatch 2023-07-14T19:21:17.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:16 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b18c85a-6628-4f12-b07f-d292fcd3a05b"}]': finished 2023-07-14T19:21:17.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:16 smithi104 ceph-mon[98435]: osdmap e45: 8 total, 7 up, 8 in 2023-07-14T19:21:17.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:16 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:17.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:16 smithi104 ceph-mon[98435]: from='client.? 172.21.15.70:0/2382433156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:21:17.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:16 smithi053 ceph-mon[96565]: from='client.? 172.21.15.70:0/4212596606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b18c85a-6628-4f12-b07f-d292fcd3a05b"}]: dispatch 2023-07-14T19:21:17.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:16 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b18c85a-6628-4f12-b07f-d292fcd3a05b"}]: dispatch 2023-07-14T19:21:17.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:16 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b18c85a-6628-4f12-b07f-d292fcd3a05b"}]': finished 2023-07-14T19:21:17.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:16 smithi053 ceph-mon[96565]: osdmap e45: 8 total, 7 up, 8 in 2023-07-14T19:21:17.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:16 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:17.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:16 smithi053 ceph-mon[96565]: from='client.? 172.21.15.70:0/2382433156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:21:17.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:16 smithi070 ceph-mon[98015]: from='client.? 172.21.15.70:0/4212596606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b18c85a-6628-4f12-b07f-d292fcd3a05b"}]: dispatch 2023-07-14T19:21:17.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:16 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b18c85a-6628-4f12-b07f-d292fcd3a05b"}]: dispatch 2023-07-14T19:21:17.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:16 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b18c85a-6628-4f12-b07f-d292fcd3a05b"}]': finished 2023-07-14T19:21:17.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:16 smithi070 ceph-mon[98015]: osdmap e45: 8 total, 7 up, 8 in 2023-07-14T19:21:17.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:16 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:17.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:16 smithi070 ceph-mon[98015]: from='client.? 172.21.15.70:0/2382433156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:21:18.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:18 smithi104 ceph-mon[98435]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:18.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:18 smithi053 ceph-mon[96565]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:18.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:18 smithi070 ceph-mon[98015]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:19.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:19 smithi070 ceph-mon[98015]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:19.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:19 smithi104 ceph-mon[98435]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:19.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:19 smithi053 ceph-mon[96565]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:21.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:21 smithi070 ceph-mon[98015]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:21.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:21 smithi104 ceph-mon[98435]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:21.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:21 smithi053 ceph-mon[96565]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:23.365 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:23 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:21:23.365 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:23 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:23.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:23 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:21:23.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:23 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:23.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:23 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:21:23.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:23 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:24.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:24 smithi070 ceph-mon[98015]: Deploying daemon osd.7 on smithi070 2023-07-14T19:21:24.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:24 smithi070 ceph-mon[98015]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:24.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:24 smithi104 ceph-mon[98435]: Deploying daemon osd.7 on smithi070 2023-07-14T19:21:24.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:24 smithi104 ceph-mon[98435]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:24.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:24 smithi053 ceph-mon[96565]: Deploying daemon osd.7 on smithi070 2023-07-14T19:21:24.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:24 smithi053 ceph-mon[96565]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:25.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:25 smithi070 ceph-mon[98015]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:25.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:25 smithi104 ceph-mon[98435]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:25.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:25 smithi053 ceph-mon[96565]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:27.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:27.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:27.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:27 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:27.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:27 smithi070 ceph-mon[98015]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:28.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:28.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:28.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:27 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:28.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:27 smithi104 ceph-mon[98435]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:28.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:28.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:28.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:27 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:28.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:27 smithi053 ceph-mon[96565]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:29.552 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.550+0000 7f6eeffff700 1 -- 172.21.15.70:0/945694948 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6ef8060a80 con 0x7f6ee40a1150 2023-07-14T19:21:29.552 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 7 on host 'smithi070' 2023-07-14T19:21:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6ee40a1150 msgr2=0x7f6ee40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6ee40a1150 0x7f6ee40a3600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0005950 tx=0x7f6ee00058e0 comp rx=0 tx=0).stop 2023-07-14T19:21:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80fee50 msgr2=0x7f6ef80fe8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80fee50 0x7f6ef80fe8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800d980 tx=0x7f6ee800dd40 comp rx=0 tx=0).stop 2023-07-14T19:21:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 shutdown_connections 2023-07-14T19:21:29.554 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6ee40a1150 0x7f6ee40a3600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ef8100280 0x7f6ef819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ef80fee50 0x7f6ef80fe8c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 --2- 172.21.15.70:0/945694948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ef80ff800 0x7f6ef819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 >> 172.21.15.70:0/945694948 conn(0x7f6ef80f99b0 msgr2=0x7f6ef81032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:21:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 shutdown_connections 2023-07-14T19:21:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:21:29.552+0000 7f6efd2fb700 1 -- 172.21.15.70:0/945694948 wait complete. 2023-07-14T19:21:29.928 DEBUG:teuthology.orchestra.run.smithi070:osd.7> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.7.service 2023-07-14T19:21:29.930 INFO:tasks.cephadm:Deploying osd.8 on smithi104 with /dev/vg_nvme/lv_2... 2023-07-14T19:21:29.931 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:21:29.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:29.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:29.956 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:29.956 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:29.956 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:29.956 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:29 smithi070 ceph-mon[98015]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:29.957 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:29.957 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:29.957 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:29.959 INFO:journalctl@ceph.osd.7.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:21:30.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:30.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:30.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:29 smithi053 ceph-mon[96565]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:30.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:30.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:30.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:30.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:29 smithi104 ceph-mon[98435]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:30.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:30.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:30.491 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:21:30.697 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:21:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:21:30.626+0000 7f3cf71f7640 -1 osd.7 0 log_to_monitors true 2023-07-14T19:21:30.976 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:30 smithi104 ceph-mon[98435]: from='osd.7 [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:21:30.977 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:30 smithi104 ceph-mon[98435]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:21:30.977 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:30 smithi070 ceph-mon[98015]: from='osd.7 [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:21:30.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:30 smithi070 ceph-mon[98015]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:21:31.071 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:30 smithi053 ceph-mon[96565]: from='osd.7 [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:21:31.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:30 smithi053 ceph-mon[96565]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:21:32.006 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:21:32.007 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:21:32.007 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2023-07-14T19:21:32.008 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2023-07-14T19:21:32.008 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0156734 s, 669 MB/s 2023-07-14T19:21:32.008 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2023-07-14T19:21:32.025 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:31 smithi070 ceph-mon[98015]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:21:32.025 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:31 smithi070 ceph-mon[98015]: osdmap e46: 8 total, 7 up, 8 in 2023-07-14T19:21:32.025 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:31 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:32.026 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:31 smithi070 ceph-mon[98015]: from='osd.7 [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:21:32.026 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:31 smithi070 ceph-mon[98015]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:21:32.026 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:31 smithi070 ceph-mon[98015]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:32.026 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:21:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:21:31.836+0000 7f3cec06e700 -1 osd.7 0 waiting for initial osdmap 2023-07-14T19:21:32.027 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:21:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:21:31.846+0000 7f3ce565f700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:21:32.135 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:31 smithi104 ceph-mon[98435]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:21:32.135 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:31 smithi104 ceph-mon[98435]: osdmap e46: 8 total, 7 up, 8 in 2023-07-14T19:21:32.136 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:31 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:32.136 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:31 smithi104 ceph-mon[98435]: from='osd.7 [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:21:32.136 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:31 smithi104 ceph-mon[98435]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:21:32.136 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:31 smithi104 ceph-mon[98435]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:32.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:31 smithi053 ceph-mon[96565]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:21:32.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:31 smithi053 ceph-mon[96565]: osdmap e46: 8 total, 7 up, 8 in 2023-07-14T19:21:32.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:31 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:32.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:31 smithi053 ceph-mon[96565]: from='osd.7 [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:21:32.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:31 smithi053 ceph-mon[96565]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:21:32.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:31 smithi053 ceph-mon[96565]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:21:32.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2023-07-14T19:21:32.893 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:21:33.099 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:21:33.099 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:21:33.099 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:33.100 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:33.100 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi070 2023-07-14T19:21:33.100 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.100 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.100 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:33.100 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:33.100 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:33.101 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi070 to 5722M 2023-07-14T19:21:33.101 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.101 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:33.101 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:33.101 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.101 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: osd.7 [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] boot 2023-07-14T19:21:33.101 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: osdmap e48: 8 total, 8 up, 8 in 2023-07-14T19:21:33.102 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:33.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:21:33.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:21:33.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:33.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:33.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi070 2023-07-14T19:21:33.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:33.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:33.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:33.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi070 to 5722M 2023-07-14T19:21:33.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:33.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:33.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: osd.7 [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] boot 2023-07-14T19:21:33.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: osdmap e48: 8 total, 8 up, 8 in 2023-07-14T19:21:33.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:33.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:21:33.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:21:33.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:33.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:33.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi070 2023-07-14T19:21:33.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi070 to 5722M 2023-07-14T19:21:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:33.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:33.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:33.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: osd.7 [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] boot 2023-07-14T19:21:33.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: osdmap e48: 8 total, 8 up, 8 in 2023-07-14T19:21:33.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:21:34.054 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.053+0000 7fca2513f700 1 -- 172.21.15.104:0/961638453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200febb0 msgr2=0x7fca200fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:34.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.053+0000 7fca2513f700 1 --2- 172.21.15.104:0/961638453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200febb0 0x7fca200fefc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fca10009990 tx=0x7fca10009ca0 comp rx=0 tx=0).stop 2023-07-14T19:21:34.055 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.053+0000 7fca2513f700 1 -- 172.21.15.104:0/961638453 shutdown_connections 2023-07-14T19:21:34.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.053+0000 7fca2513f700 1 --2- 172.21.15.104:0/961638453 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca20100740 0x7fca20107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:34.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.053+0000 7fca2513f700 1 --2- 172.21.15.104:0/961638453 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200febb0 0x7fca200fefc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:34.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.053+0000 7fca2513f700 1 --2- 172.21.15.104:0/961638453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca200ffdb0 0x7fca20100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:34.056 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.053+0000 7fca2513f700 1 -- 172.21.15.104:0/961638453 >> 172.21.15.104:0/961638453 conn(0x7fca200fa1e0 msgr2=0x7fca200fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:21:34.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.053+0000 7fca2513f700 1 -- 172.21.15.104:0/961638453 shutdown_connections 2023-07-14T19:21:34.057 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.053+0000 7fca2513f700 1 -- 172.21.15.104:0/961638453 wait complete. 2023-07-14T19:21:34.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.054+0000 7fca2513f700 1 Processor -- start 2023-07-14T19:21:34.061 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.054+0000 7fca2513f700 1 -- start start 2023-07-14T19:21:34.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.054+0000 7fca2513f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca200febb0 0x7fca201949e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:34.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.054+0000 7fca2513f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200ffdb0 0x7fca20194f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:34.062 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.054+0000 7fca2513f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca20100740 0x7fca2019bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:34.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.054+0000 7fca2513f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2019c570 con 0x7fca200febb0 2023-07-14T19:21:34.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.054+0000 7fca2513f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2019c6b0 con 0x7fca20100740 2023-07-14T19:21:34.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.054+0000 7fca2513f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2019c840 con 0x7fca200ffdb0 2023-07-14T19:21:34.063 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200ffdb0 0x7fca20194f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:34.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200ffdb0 0x7fca20194f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57108/0 (socket says 172.21.15.104:57108) 2023-07-14T19:21:34.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1e59c700 1 -- 172.21.15.104:0/1723441044 learned_addr learned my addr 172.21.15.104:0/1723441044 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:21:34.064 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1ed9d700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca200febb0 0x7fca201949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:34.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1f59e700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca20100740 0x7fca2019bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:34.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1e59c700 1 -- 172.21.15.104:0/1723441044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca20100740 msgr2=0x7fca2019bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:34.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1e59c700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca20100740 0x7fca2019bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:34.065 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1e59c700 1 -- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca200febb0 msgr2=0x7fca201949e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:34.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1e59c700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca200febb0 0x7fca201949e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:34.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1e59c700 1 -- 172.21.15.104:0/1723441044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca100095f0 con 0x7fca200ffdb0 2023-07-14T19:21:34.066 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca1e59c700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200ffdb0 0x7fca20194f20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca0800b560 tx=0x7fca0800b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:21:34.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca17fff700 1 -- 172.21.15.104:0/1723441044 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08011830 con 0x7fca200ffdb0 2023-07-14T19:21:34.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2019cb20 con 0x7fca200ffdb0 2023-07-14T19:21:34.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca17fff700 1 -- 172.21.15.104:0/1723441044 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca08011e50 con 0x7fca200ffdb0 2023-07-14T19:21:34.067 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca17fff700 1 -- 172.21.15.104:0/1723441044 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0800f2e0 con 0x7fca200ffdb0 2023-07-14T19:21:34.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.055+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2019d070 con 0x7fca200ffdb0 2023-07-14T19:21:34.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.056+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca20106200 con 0x7fca200ffdb0 2023-07-14T19:21:34.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.057+0000 7fca17fff700 1 -- 172.21.15.104:0/1723441044 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fca08011990 con 0x7fca200ffdb0 2023-07-14T19:21:34.068 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.058+0000 7fca17fff700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fca0c0a1150 0x7fca0c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:34.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.058+0000 7fca17fff700 1 -- 172.21.15.104:0/1723441044 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5426+0+0 (secure 0 0 0) 0x7fca080cb8b0 con 0x7fca200ffdb0 2023-07-14T19:21:34.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.058+0000 7fca1ed9d700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fca0c0a1150 0x7fca0c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:34.069 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.058+0000 7fca1ed9d700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fca0c0a1150 0x7fca0c0a3600 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fca10006010 tx=0x7fca1000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:21:34.070 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.062+0000 7fca17fff700 1 -- 172.21.15.104:0/1723441044 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fca08095b40 con 0x7fca200ffdb0 2023-07-14T19:21:34.258 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:34.257+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fca20060a80 con 0x7fca0c0a1150 2023-07-14T19:21:34.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:34 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:21:34.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:34 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:21:34.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:34 smithi104 ceph-mon[98435]: osdmap e49: 8 total, 8 up, 8 in 2023-07-14T19:21:34.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:34 smithi104 ceph-mon[98435]: pgmap v155: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:34 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:21:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:34 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:21:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:34 smithi053 ceph-mon[96565]: osdmap e49: 8 total, 8 up, 8 in 2023-07-14T19:21:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:34 smithi053 ceph-mon[96565]: pgmap v155: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:34.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:34 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:21:34.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:34 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:21:34.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:34 smithi070 ceph-mon[98015]: osdmap e49: 8 total, 8 up, 8 in 2023-07-14T19:21:34.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:34 smithi070 ceph-mon[98015]: pgmap v155: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:35.326 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:35 smithi104 ceph-mon[98435]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:21:35.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:35 smithi104 ceph-mon[98435]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:21:35.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:21:35.327 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:21:35.328 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:35 smithi053 ceph-mon[96565]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:21:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:35 smithi053 ceph-mon[96565]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:21:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:21:35.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:21:35.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:35 smithi070 ceph-mon[98015]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:21:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:35 smithi070 ceph-mon[98015]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:21:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:21:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:21:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:36 smithi053 ceph-mon[96565]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:21:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:36 smithi053 ceph-mon[96565]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:36.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:36 smithi070 ceph-mon[98015]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:21:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:36 smithi070 ceph-mon[98015]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:36.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:36 smithi104 ceph-mon[98435]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:21:36.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:36 smithi104 ceph-mon[98435]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:37 smithi053 ceph-mon[96565]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:21:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:37 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/3120143559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "521e5c70-e2f7-45f9-81fc-93bc6401c6b4"}]: dispatch 2023-07-14T19:21:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:37 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "521e5c70-e2f7-45f9-81fc-93bc6401c6b4"}]: dispatch 2023-07-14T19:21:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:37 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "521e5c70-e2f7-45f9-81fc-93bc6401c6b4"}]': finished 2023-07-14T19:21:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:37 smithi053 ceph-mon[96565]: osdmap e53: 9 total, 8 up, 9 in 2023-07-14T19:21:37.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:37 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:37.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:37 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/3387492915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:21:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:37 smithi070 ceph-mon[98015]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:21:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:37 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/3120143559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "521e5c70-e2f7-45f9-81fc-93bc6401c6b4"}]: dispatch 2023-07-14T19:21:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:37 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "521e5c70-e2f7-45f9-81fc-93bc6401c6b4"}]: dispatch 2023-07-14T19:21:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:37 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "521e5c70-e2f7-45f9-81fc-93bc6401c6b4"}]': finished 2023-07-14T19:21:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:37 smithi070 ceph-mon[98015]: osdmap e53: 9 total, 8 up, 9 in 2023-07-14T19:21:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:37 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:37.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:37 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/3387492915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:21:37.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:37 smithi104 ceph-mon[98435]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:21:37.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:37 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/3120143559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "521e5c70-e2f7-45f9-81fc-93bc6401c6b4"}]: dispatch 2023-07-14T19:21:37.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:37 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "521e5c70-e2f7-45f9-81fc-93bc6401c6b4"}]: dispatch 2023-07-14T19:21:37.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:37 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "521e5c70-e2f7-45f9-81fc-93bc6401c6b4"}]': finished 2023-07-14T19:21:37.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:37 smithi104 ceph-mon[98435]: osdmap e53: 9 total, 8 up, 9 in 2023-07-14T19:21:37.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:37 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:37.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:37 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/3387492915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:21:38.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:38 smithi053 ceph-mon[96565]: pgmap v161: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:38.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:38 smithi070 ceph-mon[98015]: pgmap v161: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:38.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:38 smithi104 ceph-mon[98435]: pgmap v161: 1 pgs: 1 remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:39.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:39 smithi070 ceph-mon[98015]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:39.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:39 smithi104 ceph-mon[98435]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:39.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:39 smithi053 ceph-mon[96565]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:41.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:41 smithi104 ceph-mon[98435]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:41.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:41 smithi070 ceph-mon[98015]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:41.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:41 smithi053 ceph-mon[96565]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:43.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:43 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:21:43.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:43 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:43.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:43 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:21:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:43 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:43.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:43 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:21:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:43 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:44.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:44 smithi104 ceph-mon[98435]: Deploying daemon osd.8 on smithi104 2023-07-14T19:21:44.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:44 smithi104 ceph-mon[98435]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:44.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:44 smithi053 ceph-mon[96565]: Deploying daemon osd.8 on smithi104 2023-07-14T19:21:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:44 smithi053 ceph-mon[96565]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:44.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:44 smithi070 ceph-mon[98015]: Deploying daemon osd.8 on smithi104 2023-07-14T19:21:44.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:44 smithi070 ceph-mon[98015]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:45.707 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:45 smithi104 ceph-mon[98435]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:45.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:45 smithi070 ceph-mon[98015]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:45.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:45 smithi053 ceph-mon[96565]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:48.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:48 smithi104 ceph-mon[98435]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:48.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:48.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:48.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:48 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:48.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:48 smithi053 ceph-mon[96565]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:48.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:48.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:48.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:48 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:48 smithi070 ceph-mon[98015]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:48.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:48 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:49.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:49 smithi070 ceph-mon[98015]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:49.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:49 smithi104 ceph-mon[98435]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:49.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:49 smithi053 ceph-mon[96565]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:50.273 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.272+0000 7fca17fff700 1 -- 172.21.15.104:0/1723441044 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fca20060a80 con 0x7fca0c0a1150 2023-07-14T19:21:50.273 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 8 on host 'smithi104' 2023-07-14T19:21:50.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fca0c0a1150 msgr2=0x7fca0c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:50.275 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fca0c0a1150 0x7fca0c0a3600 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fca10006010 tx=0x7fca1000b540 comp rx=0 tx=0).stop 2023-07-14T19:21:50.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200ffdb0 msgr2=0x7fca20194f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:50.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200ffdb0 0x7fca20194f20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fca0800b560 tx=0x7fca0800b920 comp rx=0 tx=0).stop 2023-07-14T19:21:50.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 shutdown_connections 2023-07-14T19:21:50.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fca0c0a1150 0x7fca0c0a3600 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:50.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca20100740 0x7fca2019bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:50.276 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca200febb0 0x7fca201949e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:50.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 --2- 172.21.15.104:0/1723441044 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fca200ffdb0 0x7fca20194f20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:50.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 >> 172.21.15.104:0/1723441044 conn(0x7fca200fa1e0 msgr2=0x7fca20103340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:21:50.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 shutdown_connections 2023-07-14T19:21:50.277 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:21:50.274+0000 7fca2513f700 1 -- 172.21.15.104:0/1723441044 wait complete. 2023-07-14T19:21:50.680 DEBUG:teuthology.orchestra.run.smithi104:osd.8> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.8.service 2023-07-14T19:21:50.684 INFO:tasks.cephadm:Deploying osd.9 on smithi053 with /dev/vg_nvme/lv_1... 2023-07-14T19:21:50.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:21:50.715 INFO:journalctl@ceph.osd.8.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:21:51.010 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:51.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:51.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:51.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:51.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:51.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:51.227 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:51.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:51.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:51.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:21:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:21:51.760 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:21:51 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:21:51.496+0000 7f9dbc96b640 -1 osd.8 0 log_to_monitors true 2023-07-14T19:21:52.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:51 smithi053 ceph-mon[96565]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:52.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:51 smithi053 ceph-mon[96565]: from='osd.8 [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:21:52.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:51 smithi053 ceph-mon[96565]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:21:52.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:51 smithi070 ceph-mon[98015]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:52.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:51 smithi070 ceph-mon[98015]: from='osd.8 [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:21:52.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:51 smithi070 ceph-mon[98015]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:21:52.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:51 smithi104 ceph-mon[98435]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:52.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:51 smithi104 ceph-mon[98435]: from='osd.8 [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:21:52.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:51 smithi104 ceph-mon[98435]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:21:52.749 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:21:52.750 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:21:52.750 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2023-07-14T19:21:52.750 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2023-07-14T19:21:52.750 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00831349 s, 1.3 GB/s 2023-07-14T19:21:52.750 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2023-07-14T19:21:53.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:52 smithi053 ceph-mon[96565]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:21:53.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:52 smithi053 ceph-mon[96565]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:21:53.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:53.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:52 smithi053 ceph-mon[96565]: from='osd.8 [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:21:53.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:52 smithi053 ceph-mon[96565]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:21:53.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi053:vg_nvme/lv_1 2023-07-14T19:21:53.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:52 smithi070 ceph-mon[98015]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:21:53.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:52 smithi070 ceph-mon[98015]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:21:53.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:53.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:52 smithi070 ceph-mon[98015]: from='osd.8 [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:21:53.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:52 smithi070 ceph-mon[98015]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:21:53.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:21:52 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:21:52.914+0000 7f9db17e2700 -1 osd.8 0 waiting for initial osdmap 2023-07-14T19:21:53.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:21:52 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:21:52.926+0000 7f9daadd3700 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:21:53.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:52 smithi104 ceph-mon[98435]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:21:53.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:52 smithi104 ceph-mon[98435]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:21:53.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:53.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:52 smithi104 ceph-mon[98435]: from='osd.8 [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:21:53.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:52 smithi104 ceph-mon[98435]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:21:53.687 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:21:54.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:21:54.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:21:54.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:54.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:54.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi104 2023-07-14T19:21:54.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:54.231 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:54.232 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:54.232 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi104 to 5722M 2023-07-14T19:21:54.232 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.232 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:54.233 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:54.233 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.233 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:53 smithi070 ceph-mon[98015]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:54.242 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:21:54.242 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:21:54.242 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:54.242 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:54.242 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi104 2023-07-14T19:21:54.243 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.243 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.243 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:54.243 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:54.243 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:54.243 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi104 to 5722M 2023-07-14T19:21:54.243 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.244 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:54.244 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:54.244 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.244 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:53 smithi053 ceph-mon[96565]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:54.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:21:54.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:21:54.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:54.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:54.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi104 2023-07-14T19:21:54.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:54.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:54.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:21:54.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi104 to 5722M 2023-07-14T19:21:54.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:54.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:21:54.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:21:54.264 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:53 smithi104 ceph-mon[98435]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:21:54.900 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.898+0000 7fb1a527d700 1 -- 172.21.15.53:0/826797507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 msgr2=0x7fb1a01070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:54.900 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.898+0000 7fb1a527d700 1 --2- 172.21.15.53:0/826797507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 0x7fb1a01070c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb19800b0a0 tx=0x7fb19800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:21:54.900 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.898+0000 7fb1a527d700 1 -- 172.21.15.53:0/826797507 shutdown_connections 2023-07-14T19:21:54.900 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.898+0000 7fb1a527d700 1 --2- 172.21.15.53:0/826797507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 0x7fb1a01070c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:54.901 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.898+0000 7fb1a527d700 1 --2- 172.21.15.53:0/826797507 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a00febd0 0x7fb1a00fefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:54.901 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.898+0000 7fb1a527d700 1 --2- 172.21.15.53:0/826797507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1a00ffe70 0x7fb1a01002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:54.901 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.898+0000 7fb1a527d700 1 -- 172.21.15.53:0/826797507 >> 172.21.15.53:0/826797507 conn(0x7fb1a00fa240 msgr2=0x7fb1a00fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:21:54.901 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.898+0000 7fb1a527d700 1 -- 172.21.15.53:0/826797507 shutdown_connections 2023-07-14T19:21:54.901 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.898+0000 7fb1a527d700 1 -- 172.21.15.53:0/826797507 wait complete. 2023-07-14T19:21:54.903 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb1a527d700 1 Processor -- start 2023-07-14T19:21:54.903 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb1a527d700 1 -- start start 2023-07-14T19:21:54.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb1a527d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1a00febd0 0x7fb1a01949b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:54.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb1a527d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a00ffe70 0x7fb1a0194ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:54.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb1a527d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 0x7fb1a019bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:54.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb1a527d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a019c540 con 0x7fb1a0100820 2023-07-14T19:21:54.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb1a527d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a019c680 con 0x7fb1a00febd0 2023-07-14T19:21:54.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb1a527d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a019c810 con 0x7fb1a00ffe70 2023-07-14T19:21:54.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb19f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 0x7fb1a019bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:54.905 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb19f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 0x7fb1a019bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49644/0 (socket says 172.21.15.53:49644) 2023-07-14T19:21:54.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb19f7fe700 1 -- 172.21.15.53:0/2433428199 learned_addr learned my addr 172.21.15.53:0/2433428199 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:21:54.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb19effd700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1a00febd0 0x7fb1a01949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:54.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.899+0000 7fb19e7fc700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a00ffe70 0x7fb1a0194ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:54.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb19effd700 1 -- 172.21.15.53:0/2433428199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a00ffe70 msgr2=0x7fb1a0194ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:54.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb19effd700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a00ffe70 0x7fb1a0194ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:54.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb19effd700 1 -- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 msgr2=0x7fb1a019bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:21:54.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb19effd700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 0x7fb1a019bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:21:54.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb19effd700 1 -- 172.21.15.53:0/2433428199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb198009d00 con 0x7fb1a00febd0 2023-07-14T19:21:54.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb19f7fe700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 0x7fb1a019bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:21:54.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb19effd700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1a00febd0 0x7fb1a01949b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb19000d750 tx=0x7fb19000da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:21:54.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb187fff700 1 -- 172.21.15.53:0/2433428199 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb190004020 con 0x7fb1a00febd0 2023-07-14T19:21:54.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a019caf0 con 0x7fb1a00febd0 2023-07-14T19:21:54.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb187fff700 1 -- 172.21.15.53:0/2433428199 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb190004830 con 0x7fb1a00febd0 2023-07-14T19:21:54.908 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb187fff700 1 -- 172.21.15.53:0/2433428199 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1900054f0 con 0x7fb1a00febd0 2023-07-14T19:21:54.908 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.900+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a019d040 con 0x7fb1a00febd0 2023-07-14T19:21:54.908 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.901+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a010a3a0 con 0x7fb1a00febd0 2023-07-14T19:21:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.903+0000 7fb187fff700 1 -- 172.21.15.53:0/2433428199 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb1900049a0 con 0x7fb1a00febd0 2023-07-14T19:21:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.903+0000 7fb187fff700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb1880a1150 0x7fb1880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:21:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.903+0000 7fb19e7fc700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb1880a1150 0x7fb1880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:21:54.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.903+0000 7fb187fff700 1 -- 172.21.15.53:0/2433428199 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5862+0+0 (secure 0 0 0) 0x7fb1900cbbd0 con 0x7fb1a00febd0 2023-07-14T19:21:54.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.904+0000 7fb19e7fc700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb1880a1150 0x7fb1880a3600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb194009be0 tx=0x7fb194009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:21:54.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:54.906+0000 7fb187fff700 1 -- 172.21.15.53:0/2433428199 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb190095cb0 con 0x7fb1a00febd0 2023-07-14T19:21:55.071 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:54 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:21:55.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:54 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:21:55.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:54 smithi053 ceph-mon[96565]: osd.8 [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] boot 2023-07-14T19:21:55.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:54 smithi053 ceph-mon[96565]: osdmap e56: 9 total, 9 up, 9 in 2023-07-14T19:21:55.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:55.118 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:21:55.117+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1a0060a80 con 0x7fb1880a1150 2023-07-14T19:21:55.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:54 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:21:55.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:54 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:21:55.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:54 smithi070 ceph-mon[98015]: osd.8 [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] boot 2023-07-14T19:21:55.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:54 smithi070 ceph-mon[98015]: osdmap e56: 9 total, 9 up, 9 in 2023-07-14T19:21:55.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:55.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:54 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:21:55.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:54 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:21:55.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:54 smithi104 ceph-mon[98435]: osd.8 [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] boot 2023-07-14T19:21:55.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:54 smithi104 ceph-mon[98435]: osdmap e56: 9 total, 9 up, 9 in 2023-07-14T19:21:55.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:21:56.216 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:55 smithi053 ceph-mon[96565]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:21:56.217 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:55 smithi053 ceph-mon[96565]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:21:56.217 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:55 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:21:56.217 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:55 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:21:56.217 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:55 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:56.217 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:55 smithi053 ceph-mon[96565]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:21:56.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:55 smithi070 ceph-mon[98015]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:21:56.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:55 smithi070 ceph-mon[98015]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:21:56.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:55 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:21:56.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:55 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:21:56.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:55 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:56.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:55 smithi070 ceph-mon[98015]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:21:56.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:55 smithi104 ceph-mon[98435]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:21:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:55 smithi104 ceph-mon[98435]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:21:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:55 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:21:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:55 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:21:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:55 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:21:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:55 smithi104 ceph-mon[98435]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:21:57.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:56 smithi053 ceph-mon[96565]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:21:57.227 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:56 smithi070 ceph-mon[98015]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:21:57.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:56 smithi104 ceph-mon[98435]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:21:58.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:57 smithi070 ceph-mon[98015]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:21:58.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:57 smithi070 ceph-mon[98015]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:21:58.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:57 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1958993073' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5795443-41f4-4230-9a33-9795ebaffe91"}]: dispatch 2023-07-14T19:21:58.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:57 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1958993073' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5795443-41f4-4230-9a33-9795ebaffe91"}]': finished 2023-07-14T19:21:58.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:57 smithi070 ceph-mon[98015]: osdmap e60: 10 total, 9 up, 10 in 2023-07-14T19:21:58.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:57 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:21:58.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:57 smithi104 ceph-mon[98435]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:21:58.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:57 smithi104 ceph-mon[98435]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:21:58.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:57 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1958993073' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5795443-41f4-4230-9a33-9795ebaffe91"}]: dispatch 2023-07-14T19:21:58.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:57 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1958993073' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5795443-41f4-4230-9a33-9795ebaffe91"}]': finished 2023-07-14T19:21:58.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:57 smithi104 ceph-mon[98435]: osdmap e60: 10 total, 9 up, 10 in 2023-07-14T19:21:58.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:57 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:21:58.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:57 smithi053 ceph-mon[96565]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:21:58.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:57 smithi053 ceph-mon[96565]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:21:58.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:57 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1958993073' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5795443-41f4-4230-9a33-9795ebaffe91"}]: dispatch 2023-07-14T19:21:58.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:57 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1958993073' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5795443-41f4-4230-9a33-9795ebaffe91"}]': finished 2023-07-14T19:21:58.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:57 smithi053 ceph-mon[96565]: osdmap e60: 10 total, 9 up, 10 in 2023-07-14T19:21:58.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:21:59.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:58 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3763526554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:21:59.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:58 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3763526554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:21:59.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:58 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3763526554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:22:00.227 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:21:59 smithi070 ceph-mon[98015]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-07-14T19:22:00.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:21:59 smithi104 ceph-mon[98435]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-07-14T19:22:00.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:21:59 smithi053 ceph-mon[96565]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 132 KiB/s, 0 objects/s recovering 2023-07-14T19:22:01.503 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:01 smithi053 ceph-mon[96565]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:22:01.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:01 smithi070 ceph-mon[98015]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:22:01.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:01 smithi104 ceph-mon[98435]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:22:03.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:03 smithi053 ceph-mon[96565]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-14T19:22:03.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:03 smithi070 ceph-mon[98015]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-14T19:22:03.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:03 smithi104 ceph-mon[98435]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2023-07-14T19:22:04.725 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:04 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:22:04.725 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:04 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:04.726 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:04 smithi053 ceph-mon[96565]: Deploying daemon osd.9 on smithi053 2023-07-14T19:22:04.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:04 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:22:04.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:04 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:04.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:04 smithi070 ceph-mon[98015]: Deploying daemon osd.9 on smithi053 2023-07-14T19:22:04.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:04 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:22:04.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:04 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:04.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:04 smithi104 ceph-mon[98435]: Deploying daemon osd.9 on smithi053 2023-07-14T19:22:05.717 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:05 smithi053 ceph-mon[96565]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-07-14T19:22:05.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:05 smithi070 ceph-mon[98015]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-07-14T19:22:05.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:05 smithi104 ceph-mon[98435]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2023-07-14T19:22:08.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:08 smithi053 ceph-mon[96565]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:22:08.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:08.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:08.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:08 smithi070 ceph-mon[98015]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:22:08.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:08 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:08.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:08 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:08.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:08 smithi104 ceph-mon[98435]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:22:08.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:08 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:08.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:08 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:09.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:09 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:09.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:09 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:09.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:09 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:10.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:10 smithi053 ceph-mon[96565]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:22:10.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:10 smithi070 ceph-mon[98015]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:22:10.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:10 smithi104 ceph-mon[98435]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:22:10.924 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 9 on host 'smithi053' 2023-07-14T19:22:10.925 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.922+0000 7fb187fff700 1 -- 172.21.15.53:0/2433428199 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb1a0060a80 con 0x7fb1880a1150 2023-07-14T19:22:10.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb1880a1150 msgr2=0x7fb1880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:10.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb1880a1150 0x7fb1880a3600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb194009be0 tx=0x7fb194009260 comp rx=0 tx=0).stop 2023-07-14T19:22:10.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1a00febd0 msgr2=0x7fb1a01949b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:10.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1a00febd0 0x7fb1a01949b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb19000d750 tx=0x7fb19000da60 comp rx=0 tx=0).stop 2023-07-14T19:22:10.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 shutdown_connections 2023-07-14T19:22:10.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb1880a1150 0x7fb1880a3600 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:10.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a0100820 0x7fb1a019bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:10.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1a00febd0 0x7fb1a01949b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:10.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 --2- 172.21.15.53:0/2433428199 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a00ffe70 0x7fb1a0194ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:10.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 >> 172.21.15.53:0/2433428199 conn(0x7fb1a00fa240 msgr2=0x7fb1a00fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:10.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 shutdown_connections 2023-07-14T19:22:10.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:10.925+0000 7fb1a527d700 1 -- 172.21.15.53:0/2433428199 wait complete. 2023-07-14T19:22:11.287 DEBUG:teuthology.orchestra.run.smithi053:osd.9> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.9.service 2023-07-14T19:22:11.289 INFO:tasks.cephadm:Deploying osd.10 on smithi070 with /dev/vg_nvme/lv_1... 2023-07-14T19:22:11.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:22:11.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:22:11.855 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:22:12.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:11 smithi104 ceph-mon[98435]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:22:12.052 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.053 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.053 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.053 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.053 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:11 smithi070 ceph-mon[98015]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:22:12.071 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:12.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:11 smithi053 ceph-mon[96565]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:22:12.364 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:22:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:22:12.210+0000 7f5235b73640 -1 osd.9 0 log_to_monitors true 2023-07-14T19:22:13.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:12 smithi070 ceph-mon[98015]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:22:13.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:12 smithi104 ceph-mon[98435]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:22:13.308 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:22:13.309 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:22:13.309 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2023-07-14T19:22:13.309 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2023-07-14T19:22:13.309 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00809369 s, 1.3 GB/s 2023-07-14T19:22:13.309 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2023-07-14T19:22:13.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:12 smithi053 ceph-mon[96565]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:22:13.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi070:vg_nvme/lv_1 2023-07-14T19:22:13.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:22:13.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T19:22:13.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:13.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:22:13.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:22:13.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:13.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:13.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:13.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:13.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:13.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:13.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:13.980 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:13.980 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:13.980 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:14.071 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:22:13.933+0000 7f522a9ea700 -1 osd.9 0 waiting for initial osdmap 2023-07-14T19:22:14.072 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:22:13.948+0000 7f52237da700 -1 osd.9 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:22:14.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:22:14.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T19:22:14.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:14.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:22:14.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:22:14.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:14.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:14.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:14.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:14.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:14.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:14.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:14.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:14.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:14.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:14.250 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:22:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:22:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T19:22:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2023-07-14T19:22:14.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:22:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:14.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:14.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:14.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:14.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:15.209 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi053 2023-07-14T19:22:15.209 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi053 to 4293M 2023-07-14T19:22:15.210 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:22:15.210 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:22:15.210 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:15.210 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:15.210 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:15.210 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:15.210 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:15.211 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:14 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:15.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi053 2023-07-14T19:22:15.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi053 to 4293M 2023-07-14T19:22:15.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:22:15.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:22:15.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:15.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:15.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:15.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:15.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:15.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:14 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:15.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi053 2023-07-14T19:22:15.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi053 to 4293M 2023-07-14T19:22:15.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: from='osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2023-07-14T19:22:15.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:22:15.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:15.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:15.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:15.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:15.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:15.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:14 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.381+0000 7fb65695d700 1 -- 172.21.15.70:0/3441876181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500ffe70 msgr2=0x7fb6501002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:15.385 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.381+0000 7fb65695d700 1 --2- 172.21.15.70:0/3441876181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500ffe70 0x7fb6501002e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb644009990 tx=0x7fb644009ca0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.385 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.381+0000 7fb65695d700 1 -- 172.21.15.70:0/3441876181 shutdown_connections 2023-07-14T19:22:15.385 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.381+0000 7fb65695d700 1 --2- 172.21.15.70:0/3441876181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb650100820 0x7fb6501070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.385 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.381+0000 7fb65695d700 1 --2- 172.21.15.70:0/3441876181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6500febd0 0x7fb6500fefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.385 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.381+0000 7fb65695d700 1 --2- 172.21.15.70:0/3441876181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500ffe70 0x7fb6501002e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.381+0000 7fb65695d700 1 -- 172.21.15.70:0/3441876181 >> 172.21.15.70:0/3441876181 conn(0x7fb6500fa240 msgr2=0x7fb6500fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.381+0000 7fb65695d700 1 -- 172.21.15.70:0/3441876181 shutdown_connections 2023-07-14T19:22:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.381+0000 7fb65695d700 1 -- 172.21.15.70:0/3441876181 wait complete. 2023-07-14T19:22:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.382+0000 7fb65695d700 1 Processor -- start 2023-07-14T19:22:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.382+0000 7fb65695d700 1 -- start start 2023-07-14T19:22:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.382+0000 7fb65695d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500febd0 0x7fb6501958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.382+0000 7fb65695d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6500ffe70 0x7fb650195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.382+0000 7fb65695d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb650100820 0x7fb65019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.382+0000 7fb65695d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65019d470 con 0x7fb650100820 2023-07-14T19:22:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.382+0000 7fb65695d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65019d5b0 con 0x7fb6500febd0 2023-07-14T19:22:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.382+0000 7fb65695d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65019d740 con 0x7fb6500ffe70 2023-07-14T19:22:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500febd0 0x7fb6501958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500febd0 0x7fb6501958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56564/0 (socket says 172.21.15.70:56564) 2023-07-14T19:22:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64ffff700 1 -- 172.21.15.70:0/260792555 learned_addr learned my addr 172.21.15.70:0/260792555 (peer_addr_for_me v2:172.21.15.70:0/0) 2023-07-14T19:22:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb654efa700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb650100820 0x7fb65019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64f7fe700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6500ffe70 0x7fb650195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64ffff700 1 -- 172.21.15.70:0/260792555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6500ffe70 msgr2=0x7fb650195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64ffff700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6500ffe70 0x7fb650195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64ffff700 1 -- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb650100820 msgr2=0x7fb65019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64ffff700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb650100820 0x7fb65019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64ffff700 1 -- 172.21.15.70:0/260792555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440095f0 con 0x7fb6500febd0 2023-07-14T19:22:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64ffff700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500febd0 0x7fb6501958e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb64000c9b0 tx=0x7fb64000ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64d7fa700 1 -- 172.21.15.70:0/260792555 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640007b00 con 0x7fb6500febd0 2023-07-14T19:22:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64d7fa700 1 -- 172.21.15.70:0/260792555 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb64000ce50 con 0x7fb6500febd0 2023-07-14T19:22:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65019da20 con 0x7fb6500febd0 2023-07-14T19:22:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb64d7fa700 1 -- 172.21.15.70:0/260792555 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640018660 con 0x7fb6500febd0 2023-07-14T19:22:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.383+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65019df20 con 0x7fb6500febd0 2023-07-14T19:22:15.390 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.384+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb650106c40 con 0x7fb6500febd0 2023-07-14T19:22:15.390 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.385+0000 7fb64d7fa700 1 -- 172.21.15.70:0/260792555 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb64000f450 con 0x7fb6500febd0 2023-07-14T19:22:15.390 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.386+0000 7fb64d7fa700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb6380a11d0 0x7fb6380a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:15.390 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.386+0000 7fb64d7fa700 1 -- 172.21.15.70:0/260792555 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fb6400cc0b0 con 0x7fb6500febd0 2023-07-14T19:22:15.390 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.386+0000 7fb64f7fe700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb6380a11d0 0x7fb6380a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:15.391 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.387+0000 7fb64f7fe700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb6380a11d0 0x7fb6380a3680 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb64400b5c0 tx=0x7fb644005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:15.391 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.390+0000 7fb64d7fa700 1 -- 172.21.15.70:0/260792555 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb640095f60 con 0x7fb6500febd0 2023-07-14T19:22:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:15.587+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb650060a80 con 0x7fb6380a11d0 2023-07-14T19:22:16.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:15 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:22:16.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:15 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:22:16.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:15 smithi070 ceph-mon[98015]: osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] boot 2023-07-14T19:22:16.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:15 smithi070 ceph-mon[98015]: osdmap e63: 10 total, 10 up, 10 in 2023-07-14T19:22:16.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:16.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:15 smithi070 ceph-mon[98015]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:22:16.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:22:16.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:22:16.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:16.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:15 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:22:16.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:15 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:22:16.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:15 smithi104 ceph-mon[98435]: osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] boot 2023-07-14T19:22:16.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:15 smithi104 ceph-mon[98435]: osdmap e63: 10 total, 10 up, 10 in 2023-07-14T19:22:16.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:16.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:15 smithi104 ceph-mon[98435]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:22:16.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:22:16.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:22:16.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:16.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:15 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:22:16.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:15 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:22:16.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:15 smithi053 ceph-mon[96565]: osd.9 [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] boot 2023-07-14T19:22:16.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:15 smithi053 ceph-mon[96565]: osdmap e63: 10 total, 10 up, 10 in 2023-07-14T19:22:16.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:22:16.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:15 smithi053 ceph-mon[96565]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:22:16.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:22:16.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:22:16.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:17.227 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:16 smithi070 ceph-mon[98015]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:17.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:16 smithi070 ceph-mon[98015]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T19:22:17.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:16 smithi104 ceph-mon[98435]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:17.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:16 smithi104 ceph-mon[98435]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T19:22:17.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:16 smithi053 ceph-mon[96565]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:17.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:16 smithi053 ceph-mon[96565]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T19:22:18.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:18 smithi053 ceph-mon[96565]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:18.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:18 smithi053 ceph-mon[96565]: from='client.? 172.21.15.70:0/3667778753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff27354b-283d-406f-b2ea-fe1c9f769e1a"}]: dispatch 2023-07-14T19:22:18.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:18 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff27354b-283d-406f-b2ea-fe1c9f769e1a"}]: dispatch 2023-07-14T19:22:18.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:18 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff27354b-283d-406f-b2ea-fe1c9f769e1a"}]': finished 2023-07-14T19:22:18.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:18 smithi053 ceph-mon[96565]: osdmap e65: 11 total, 10 up, 11 in 2023-07-14T19:22:18.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:18 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:18.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:18 smithi070 ceph-mon[98015]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:18.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:18 smithi070 ceph-mon[98015]: from='client.? 172.21.15.70:0/3667778753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff27354b-283d-406f-b2ea-fe1c9f769e1a"}]: dispatch 2023-07-14T19:22:18.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:18 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff27354b-283d-406f-b2ea-fe1c9f769e1a"}]: dispatch 2023-07-14T19:22:18.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:18 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff27354b-283d-406f-b2ea-fe1c9f769e1a"}]': finished 2023-07-14T19:22:18.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:18 smithi070 ceph-mon[98015]: osdmap e65: 11 total, 10 up, 11 in 2023-07-14T19:22:18.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:18 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:18.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:18 smithi104 ceph-mon[98435]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:18.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:18 smithi104 ceph-mon[98435]: from='client.? 172.21.15.70:0/3667778753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff27354b-283d-406f-b2ea-fe1c9f769e1a"}]: dispatch 2023-07-14T19:22:18.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:18 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff27354b-283d-406f-b2ea-fe1c9f769e1a"}]: dispatch 2023-07-14T19:22:18.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:18 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff27354b-283d-406f-b2ea-fe1c9f769e1a"}]': finished 2023-07-14T19:22:18.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:18 smithi104 ceph-mon[98435]: osdmap e65: 11 total, 10 up, 11 in 2023-07-14T19:22:18.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:18 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:19.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:19 smithi053 ceph-mon[96565]: from='client.? 172.21.15.70:0/538267381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:22:19.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:19 smithi070 ceph-mon[98015]: from='client.? 172.21.15.70:0/538267381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:22:19.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:19 smithi104 ceph-mon[98435]: from='client.? 172.21.15.70:0/538267381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:22:20.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:20 smithi053 ceph-mon[96565]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:20.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:20 smithi070 ceph-mon[98015]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:20.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:20 smithi104 ceph-mon[98435]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:21.626 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:21 smithi070 ceph-mon[98015]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:21.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:21 smithi104 ceph-mon[98435]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:21.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:21 smithi053 ceph-mon[96565]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:23.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:23 smithi070 ceph-mon[98015]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:23.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:23 smithi104 ceph-mon[98435]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:23.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:23 smithi053 ceph-mon[96565]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:24.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:24 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:22:24.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:24 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:24.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:24 smithi070 ceph-mon[98015]: Deploying daemon osd.10 on smithi070 2023-07-14T19:22:24.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:24 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:22:24.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:24 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:24.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:24 smithi104 ceph-mon[98435]: Deploying daemon osd.10 on smithi070 2023-07-14T19:22:24.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:24 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:22:24.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:24 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:24.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:24 smithi053 ceph-mon[96565]: Deploying daemon osd.10 on smithi070 2023-07-14T19:22:25.632 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:25 smithi070 ceph-mon[98015]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:25.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:25 smithi104 ceph-mon[98435]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:25.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:25 smithi053 ceph-mon[96565]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:28.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:28 smithi070 ceph-mon[98015]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:28.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:28 smithi053 ceph-mon[96565]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:28.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:28 smithi104 ceph-mon[98435]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:29.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:29.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:29.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:29.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:29 smithi070 ceph-mon[98015]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:29.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:29.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:29.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:29.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:29 smithi104 ceph-mon[98435]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:29.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:29.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:29.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:29.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:29 smithi053 ceph-mon[96565]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:31.054 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.052+0000 7fb64d7fa700 1 -- 172.21.15.70:0/260792555 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fb650060a80 con 0x7fb6380a11d0 2023-07-14T19:22:31.054 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 10 on host 'smithi070' 2023-07-14T19:22:31.056 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.054+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb6380a11d0 msgr2=0x7fb6380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.054+0000 7fb65695d700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb6380a11d0 0x7fb6380a3680 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb64400b5c0 tx=0x7fb644005960 comp rx=0 tx=0).stop 2023-07-14T19:22:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.054+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500febd0 msgr2=0x7fb6501958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.054+0000 7fb65695d700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500febd0 0x7fb6501958e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb64000c9b0 tx=0x7fb64000ccc0 comp rx=0 tx=0).stop 2023-07-14T19:22:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.055+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 shutdown_connections 2023-07-14T19:22:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.055+0000 7fb65695d700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb6380a11d0 0x7fb6380a3680 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.055+0000 7fb65695d700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb650100820 0x7fb65019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:31.057 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.055+0000 7fb65695d700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6500febd0 0x7fb6501958e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:31.058 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.055+0000 7fb65695d700 1 --2- 172.21.15.70:0/260792555 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6500ffe70 0x7fb650195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:31.058 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.055+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 >> 172.21.15.70:0/260792555 conn(0x7fb6500fa240 msgr2=0x7fb6500fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:31.058 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.055+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 shutdown_connections 2023-07-14T19:22:31.058 INFO:teuthology.orchestra.run.smithi070.stderr:2023-07-14T19:22:31.055+0000 7fb65695d700 1 -- 172.21.15.70:0/260792555 wait complete. 2023-07-14T19:22:31.425 DEBUG:teuthology.orchestra.run.smithi070:osd.10> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.10.service 2023-07-14T19:22:31.428 INFO:tasks.cephadm:Deploying osd.11 on smithi104 with /dev/vg_nvme/lv_1... 2023-07-14T19:22:31.428 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:22:31.460 INFO:journalctl@ceph.osd.10.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:22:31.991 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:22:32.191 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.191 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.191 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.191 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.192 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:32 smithi070 ceph-mon[98015]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:32.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:32 smithi053 ceph-mon[96565]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:32.369 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.370 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.370 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.370 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:32.370 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:32 smithi104 ceph-mon[98435]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:32.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:22:32 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:22:32.392+0000 7f651cd1a640 -1 osd.10 0 log_to_monitors true 2023-07-14T19:22:33.510 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:22:33.511 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:22:33.511 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2023-07-14T19:22:33.511 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2023-07-14T19:22:33.511 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111071 s, 944 MB/s 2023-07-14T19:22:33.511 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2023-07-14T19:22:33.526 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:33 smithi070 ceph-mon[98015]: from='osd.10 [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:22:33.526 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:33 smithi070 ceph-mon[98015]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:22:33.546 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:33 smithi104 ceph-mon[98435]: from='osd.10 [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:22:33.546 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:33 smithi104 ceph-mon[98435]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:22:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:33 smithi053 ceph-mon[96565]: from='osd.10 [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:22:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:33 smithi053 ceph-mon[96565]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:22:33.945 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi104:vg_nvme/lv_1 2023-07-14T19:22:34.494 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:22:34.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:22:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T19:22:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='osd.10 [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:22:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:22:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:34.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:34.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:34.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:34.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:34.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:34.575 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:34 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:22:34.694 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T19:22:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='osd.10 [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:22:34.695 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:22:34.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:34.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.696 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.697 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:34.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:34.698 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.699 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:34.699 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:34.699 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:34.699 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:34 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.728 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:22:34.283+0000 7f6511b91700 -1 osd.10 0 waiting for initial osdmap 2023-07-14T19:22:34.728 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:22:34.295+0000 7f650c985700 -1 osd.10 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:22:34.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:22:34.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T19:22:34.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:34.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='osd.10 [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:22:34.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2023-07-14T19:22:34.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:22:34.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:34.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:34.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:34.733 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:34.733 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:34.733 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:34.733 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:34.733 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:34 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:35.551 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:35 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi070 2023-07-14T19:22:35.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:35 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi070 to 4292M 2023-07-14T19:22:35.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:35 smithi104 ceph-mon[98435]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:22:35.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:35 smithi104 ceph-mon[98435]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:22:35.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:35.552 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:35.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:35 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi070 2023-07-14T19:22:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:35 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi070 to 4292M 2023-07-14T19:22:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:35 smithi053 ceph-mon[96565]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:22:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:35 smithi053 ceph-mon[96565]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:22:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:35.697 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.693+0000 7f89955e2700 1 -- 172.21.15.104:0/398819903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900ff810 msgr2=0x7f89901070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.693+0000 7f89955e2700 1 --2- 172.21.15.104:0/398819903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900ff810 0x7f89901070c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f898800b0a0 tx=0x7f898800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.693+0000 7f89955e2700 1 -- 172.21.15.104:0/398819903 shutdown_connections 2023-07-14T19:22:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.693+0000 7f89955e2700 1 --2- 172.21.15.104:0/398819903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900ff810 0x7f89901070c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.693+0000 7f89955e2700 1 --2- 172.21.15.104:0/398819903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8990100290 0x7f89901006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.693+0000 7f89955e2700 1 --2- 172.21.15.104:0/398819903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89900fee60 0x7f89900ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.693+0000 7f89955e2700 1 -- 172.21.15.104:0/398819903 >> 172.21.15.104:0/398819903 conn(0x7f89900f99c0 msgr2=0x7f89900fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:35.698 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.693+0000 7f89955e2700 1 -- 172.21.15.104:0/398819903 shutdown_connections 2023-07-14T19:22:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.693+0000 7f89955e2700 1 -- 172.21.15.104:0/398819903 wait complete. 2023-07-14T19:22:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f89955e2700 1 Processor -- start 2023-07-14T19:22:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f89955e2700 1 -- start start 2023-07-14T19:22:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f89955e2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900fee60 0x7f8990195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f89955e2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89900ff810 0x7f89901961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f89955e2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8990100290 0x7f899019d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:35.699 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f89955e2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899019d7b0 con 0x7f8990100290 2023-07-14T19:22:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f89955e2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899019d8f0 con 0x7f89900ff810 2023-07-14T19:22:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f89955e2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899019dab0 con 0x7f89900fee60 2023-07-14T19:22:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f898effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900fee60 0x7f8990195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f898effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900fee60 0x7f8990195c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57244/0 (socket says 172.21.15.104:57244) 2023-07-14T19:22:35.700 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.694+0000 7f898effd700 1 -- 172.21.15.104:0/3035536995 learned_addr learned my addr 172.21.15.104:0/3035536995 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:22:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f898e7fc700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89900ff810 0x7f89901961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f898f7fe700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8990100290 0x7f899019d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f898effd700 1 -- 172.21.15.104:0/3035536995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89900ff810 msgr2=0x7f89901961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f898effd700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89900ff810 0x7f89901961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f898effd700 1 -- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8990100290 msgr2=0x7f899019d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:35.701 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f898effd700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8990100290 0x7f899019d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f898effd700 1 -- 172.21.15.104:0/3035536995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988009d00 con 0x7f89900fee60 2023-07-14T19:22:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f898f7fe700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8990100290 0x7f899019d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:22:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f898effd700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900fee60 0x7f8990195c60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f898000b530 tx=0x7f898000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f8977fff700 1 -- 172.21.15.104:0/3035536995 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8980011830 con 0x7f89900fee60 2023-07-14T19:22:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f8977fff700 1 -- 172.21.15.104:0/3035536995 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8980011e50 con 0x7f89900fee60 2023-07-14T19:22:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f8977fff700 1 -- 172.21.15.104:0/3035536995 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898000f2e0 con 0x7f89900fee60 2023-07-14T19:22:35.702 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899019dd90 con 0x7f89900fee60 2023-07-14T19:22:35.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.695+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899019e200 con 0x7f89900fee60 2023-07-14T19:22:35.703 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.696+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8990106d60 con 0x7f89900fee60 2023-07-14T19:22:35.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.697+0000 7f8977fff700 1 -- 172.21.15.104:0/3035536995 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8980010380 con 0x7f89900fee60 2023-07-14T19:22:35.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.699+0000 7f8977fff700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89780a1150 0x7f89780a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:35.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.699+0000 7f8977fff700 1 -- 172.21.15.104:0/3035536995 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f89800cc3d0 con 0x7f89900fee60 2023-07-14T19:22:35.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.699+0000 7f898e7fc700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89780a1150 0x7f89780a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:35.704 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.699+0000 7f898e7fc700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89780a1150 0x7f89780a3600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8984009a10 tx=0x7f8984009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:35.705 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.701+0000 7f8977fff700 1 -- 172.21.15.104:0/3035536995 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8980096150 con 0x7f89900fee60 2023-07-14T19:22:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:35 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi070 2023-07-14T19:22:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:35 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi070 to 4292M 2023-07-14T19:22:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:35 smithi070 ceph-mon[98015]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2023-07-14T19:22:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:35 smithi070 ceph-mon[98015]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:22:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:35.896 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:35.894+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8990060a80 con 0x7f89780a1150 2023-07-14T19:22:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:36 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:22:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:36 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:22:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:36 smithi053 ceph-mon[96565]: osd.10 [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] boot 2023-07-14T19:22:36.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:36 smithi053 ceph-mon[96565]: osdmap e68: 11 total, 11 up, 11 in 2023-07-14T19:22:36.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:36.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:36 smithi053 ceph-mon[96565]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:36.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:22:36.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:22:36.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:36.668 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:36 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:22:36.668 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:36 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:22:36.668 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:36 smithi104 ceph-mon[98435]: osd.10 [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] boot 2023-07-14T19:22:36.668 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:36 smithi104 ceph-mon[98435]: osdmap e68: 11 total, 11 up, 11 in 2023-07-14T19:22:36.668 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:36.668 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:36 smithi104 ceph-mon[98435]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:36.669 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:22:36.669 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:22:36.669 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:36.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:36 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:22:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:36 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:22:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:36 smithi070 ceph-mon[98015]: osd.10 [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] boot 2023-07-14T19:22:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:36 smithi070 ceph-mon[98015]: osdmap e68: 11 total, 11 up, 11 in 2023-07-14T19:22:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:22:36.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:36 smithi070 ceph-mon[98015]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:36.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:22:36.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:22:36.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:37.545 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:37 smithi104 ceph-mon[98435]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:37.545 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:37 smithi104 ceph-mon[98435]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T19:22:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:37 smithi053 ceph-mon[96565]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:37 smithi053 ceph-mon[96565]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T19:22:37.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:37 smithi070 ceph-mon[98015]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:37 smithi070 ceph-mon[98015]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T19:22:38.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:38 smithi053 ceph-mon[96565]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:22:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:38 smithi053 ceph-mon[96565]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:38 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/757172262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2d7cdf0-c8ac-4fc1-a017-e641c666619a"}]: dispatch 2023-07-14T19:22:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:38 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2d7cdf0-c8ac-4fc1-a017-e641c666619a"}]: dispatch 2023-07-14T19:22:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:38 smithi053 ceph-mon[96565]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2d7cdf0-c8ac-4fc1-a017-e641c666619a"}]': finished 2023-07-14T19:22:38.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:38 smithi053 ceph-mon[96565]: osdmap e71: 12 total, 11 up, 12 in 2023-07-14T19:22:38.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:38 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:38.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:38 smithi070 ceph-mon[98015]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:22:38.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:38 smithi070 ceph-mon[98015]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:38.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:38 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/757172262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2d7cdf0-c8ac-4fc1-a017-e641c666619a"}]: dispatch 2023-07-14T19:22:38.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:38 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2d7cdf0-c8ac-4fc1-a017-e641c666619a"}]: dispatch 2023-07-14T19:22:38.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:38 smithi070 ceph-mon[98015]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2d7cdf0-c8ac-4fc1-a017-e641c666619a"}]': finished 2023-07-14T19:22:38.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:38 smithi070 ceph-mon[98015]: osdmap e71: 12 total, 11 up, 12 in 2023-07-14T19:22:38.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:38 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:38.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:38 smithi104 ceph-mon[98435]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:22:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:38 smithi104 ceph-mon[98435]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:38 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/757172262' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2d7cdf0-c8ac-4fc1-a017-e641c666619a"}]: dispatch 2023-07-14T19:22:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:38 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2d7cdf0-c8ac-4fc1-a017-e641c666619a"}]: dispatch 2023-07-14T19:22:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:38 smithi104 ceph-mon[98435]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2d7cdf0-c8ac-4fc1-a017-e641c666619a"}]': finished 2023-07-14T19:22:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:38 smithi104 ceph-mon[98435]: osdmap e71: 12 total, 11 up, 12 in 2023-07-14T19:22:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:38 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:39 smithi053 ceph-mon[96565]: from='client.? 172.21.15.104:0/696817280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:22:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:39 smithi053 ceph-mon[96565]: osdmap e72: 12 total, 11 up, 12 in 2023-07-14T19:22:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:39.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:39 smithi070 ceph-mon[98015]: from='client.? 172.21.15.104:0/696817280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:22:39.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:39 smithi070 ceph-mon[98015]: osdmap e72: 12 total, 11 up, 12 in 2023-07-14T19:22:39.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:39.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:39 smithi104 ceph-mon[98435]: from='client.? 172.21.15.104:0/696817280' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:22:39.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:39 smithi104 ceph-mon[98435]: osdmap e72: 12 total, 11 up, 12 in 2023-07-14T19:22:39.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:40.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:40 smithi053 ceph-mon[96565]: pgmap v211: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:22:40.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:40 smithi070 ceph-mon[98015]: pgmap v211: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:22:40.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:40 smithi104 ceph-mon[98435]: pgmap v211: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:22:41.571 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:22:41 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:22:41.273+0000 7f4dc37d1700 -1 calc_pg_upmaps abort due to max <= 0 2023-07-14T19:22:42.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:42 smithi104 ceph-mon[98435]: pgmap v212: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-14T19:22:42.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:42 smithi053 ceph-mon[96565]: pgmap v212: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-14T19:22:42.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:42 smithi070 ceph-mon[98015]: pgmap v212: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-14T19:22:44.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:44 smithi104 ceph-mon[98435]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:22:44.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:44 smithi053 ceph-mon[96565]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:22:44.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:44 smithi070 ceph-mon[98015]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:22:45.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:45 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:22:45.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:45 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:45.608 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:45 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:22:45.608 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:45 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:45.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:45 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:22:45.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:45 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:46 smithi053 ceph-mon[96565]: Deploying daemon osd.11 on smithi104 2023-07-14T19:22:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:46 smithi053 ceph-mon[96565]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:22:46.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:46 smithi070 ceph-mon[98015]: Deploying daemon osd.11 on smithi104 2023-07-14T19:22:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:46 smithi070 ceph-mon[98015]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:22:46.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:46 smithi104 ceph-mon[98435]: Deploying daemon osd.11 on smithi104 2023-07-14T19:22:46.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:46 smithi104 ceph-mon[98435]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:22:48.528 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:48 smithi104 ceph-mon[98435]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:48.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:48 smithi053 ceph-mon[96565]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:48 smithi070 ceph-mon[98015]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:49.443 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:49 smithi104 ceph-mon[98435]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:49.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:49 smithi070 ceph-mon[98015]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:49.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:49 smithi053 ceph-mon[96565]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:50.602 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:50.602 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:50.603 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:50 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:50.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:50.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:50.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:50 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:50.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:50.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:50.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:50 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:51.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:51 smithi104 ceph-mon[98435]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:51.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:51 smithi053 ceph-mon[96565]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:51.977 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:51 smithi070 ceph-mon[98015]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:52.211 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 11 on host 'smithi104' 2023-07-14T19:22:52.212 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.209+0000 7f8977fff700 1 -- 172.21.15.104:0/3035536995 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f8990060a80 con 0x7f89780a1150 2023-07-14T19:22:52.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89780a1150 msgr2=0x7f89780a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:52.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89780a1150 0x7f89780a3600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8984009a10 tx=0x7f8984009190 comp rx=0 tx=0).stop 2023-07-14T19:22:52.213 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900fee60 msgr2=0x7f8990195c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:52.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900fee60 0x7f8990195c60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f898000b530 tx=0x7f898000b840 comp rx=0 tx=0).stop 2023-07-14T19:22:52.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 shutdown_connections 2023-07-14T19:22:52.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89780a1150 0x7f89780a3600 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:52.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8990100290 0x7f899019d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:52.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89900fee60 0x7f8990195c60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:52.214 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 --2- 172.21.15.104:0/3035536995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89900ff810 0x7f89901961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:52.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 >> 172.21.15.104:0/3035536995 conn(0x7f89900f99c0 msgr2=0x7f89900fbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:52.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 shutdown_connections 2023-07-14T19:22:52.215 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:22:52.212+0000 7f89955e2700 1 -- 172.21.15.104:0/3035536995 wait complete. 2023-07-14T19:22:52.574 DEBUG:teuthology.orchestra.run.smithi104:osd.11> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.11.service 2023-07-14T19:22:52.576 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-14T19:22:52.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T19:22:52.605 INFO:journalctl@ceph.osd.11.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:22:53.154 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:22:53.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:22:53 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:22:53.159+0000 7fb7c1911640 -1 osd.11 0 log_to_monitors true 2023-07-14T19:22:53.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:53.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:53.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:52 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:53.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:53.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:53.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:52 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:53.477 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:53.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:53.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:53.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:52 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:54.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:53 smithi104 ceph-mon[98435]: from='osd.11 [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:22:54.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:53 smithi104 ceph-mon[98435]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:22:54.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:53 smithi104 ceph-mon[98435]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:54.282 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:53 smithi053 ceph-mon[96565]: from='osd.11 [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:22:54.283 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:53 smithi053 ceph-mon[96565]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:22:54.283 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:53 smithi053 ceph-mon[96565]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 -- 172.21.15.53:0/1672573641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40ffdb0 msgr2=0x7fcfa4100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 --2- 172.21.15.53:0/1672573641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40ffdb0 0x7fcfa4100200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009990 tx=0x7fcf90009ca0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 -- 172.21.15.53:0/1672573641 shutdown_connections 2023-07-14T19:22:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 --2- 172.21.15.53:0/1672573641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfa4100740 0x7fcfa4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 --2- 172.21.15.53:0/1672573641 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfa40febb0 0x7fcfa40fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 --2- 172.21.15.53:0/1672573641 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40ffdb0 0x7fcfa4100200 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 -- 172.21.15.53:0/1672573641 >> 172.21.15.53:0/1672573641 conn(0x7fcfa40fa1e0 msgr2=0x7fcfa40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 -- 172.21.15.53:0/1672573641 shutdown_connections 2023-07-14T19:22:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 -- 172.21.15.53:0/1672573641 wait complete. 2023-07-14T19:22:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 Processor -- start 2023-07-14T19:22:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.463+0000 7fcfab20d700 1 -- start start 2023-07-14T19:22:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfab20d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40febb0 0x7fcfa4069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfab20d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfa40ffdb0 0x7fcfa419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfab20d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfa4100740 0x7fcfa419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfab20d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa419d5b0 con 0x7fcfa40febb0 2023-07-14T19:22:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfab20d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa419d720 con 0x7fcfa40ffdb0 2023-07-14T19:22:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfab20d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa419d8e0 con 0x7fcfa4100740 2023-07-14T19:22:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa8fa9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40febb0 0x7fcfa4069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa8fa9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40febb0 0x7fcfa4069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49812/0 (socket says 172.21.15.53:49812) 2023-07-14T19:22:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa8fa9700 1 -- 172.21.15.53:0/2940104625 learned_addr learned my addr 172.21.15.53:0/2940104625 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:22:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa97aa700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfa4100740 0x7fcfa419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa3fff700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfa40ffdb0 0x7fcfa419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa8fa9700 1 -- 172.21.15.53:0/2940104625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfa4100740 msgr2=0x7fcfa419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa8fa9700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfa4100740 0x7fcfa419ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa8fa9700 1 -- 172.21.15.53:0/2940104625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfa40ffdb0 msgr2=0x7fcfa419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa8fa9700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfa40ffdb0 0x7fcfa419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa8fa9700 1 -- 172.21.15.53:0/2940104625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf900095f0 con 0x7fcfa40febb0 2023-07-14T19:22:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa3fff700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfa40ffdb0 0x7fcfa419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:22:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa8fa9700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40febb0 0x7fcfa4069510 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800b500 tx=0x7fcf9800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa1ffb700 1 -- 172.21.15.53:0/2940104625 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98011830 con 0x7fcfa40febb0 2023-07-14T19:22:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa1ffb700 1 -- 172.21.15.53:0/2940104625 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf98011e50 con 0x7fcfa40febb0 2023-07-14T19:22:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa419dbc0 con 0x7fcfa40febb0 2023-07-14T19:22:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.464+0000 7fcfa1ffb700 1 -- 172.21.15.53:0/2940104625 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800f2e0 con 0x7fcfa40febb0 2023-07-14T19:22:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.465+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa419e0c0 con 0x7fcfa40febb0 2023-07-14T19:22:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.466+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa41061a0 con 0x7fcfa40febb0 2023-07-14T19:22:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.467+0000 7fcfa1ffb700 1 -- 172.21.15.53:0/2940104625 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcf98010580 con 0x7fcfa40febb0 2023-07-14T19:22:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.467+0000 7fcfa1ffb700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcf940a1150 0x7fcf940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.467+0000 7fcfa3fff700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcf940a1150 0x7fcf940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.467+0000 7fcfa1ffb700 1 -- 172.21.15.53:0/2940104625 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fcf980cbe10 con 0x7fcfa40febb0 2023-07-14T19:22:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.468+0000 7fcfa3fff700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcf940a1150 0x7fcf940a3600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000b5c0 tx=0x7fcf90005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.472+0000 7fcfa1ffb700 1 -- 172.21.15.53:0/2940104625 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcf98095b30 con 0x7fcfa40febb0 2023-07-14T19:22:54.477 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:53 smithi070 ceph-mon[98015]: from='osd.11 [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:22:54.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:53 smithi070 ceph-mon[98015]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:22:54.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:53 smithi070 ceph-mon[98015]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:54.670 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.668+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fcfa402cb30 con 0x7fcfa40febb0 2023-07-14T19:22:54.671 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.668+0000 7fcfa1ffb700 1 -- 172.21.15.53:0/2940104625 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) v1 ==== 74+0+133 (secure 0 0 0) 0x7fcf98099150 con 0x7fcfa40febb0 2023-07-14T19:22:54.671 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:22:54.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcf940a1150 msgr2=0x7fcf940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:54.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcf940a1150 0x7fcf940a3600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000b5c0 tx=0x7fcf90005fb0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40febb0 msgr2=0x7fcfa4069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:54.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40febb0 0x7fcfa4069510 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800b500 tx=0x7fcf9800b8c0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 shutdown_connections 2023-07-14T19:22:54.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcf940a1150 0x7fcf940a3600 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfa4100740 0x7fcfa419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfa40febb0 0x7fcfa4069510 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 --2- 172.21.15.53:0/2940104625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfa40ffdb0 0x7fcfa419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:54.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 >> 172.21.15.53:0/2940104625 conn(0x7fcfa40fa1e0 msgr2=0x7fcfa41032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:54.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.670+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 shutdown_connections 2023-07-14T19:22:54.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:54.671+0000 7fcfab20d700 1 -- 172.21.15.53:0/2940104625 wait complete. 2023-07-14T19:22:55.099 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":73,"num_osds":12,"num_up_osds":11,"osd_up_since":1689362555,"num_in_osds":12,"osd_in_since":1689362558,"num_remapped_pgs":0} 2023-07-14T19:22:55.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:22:54.994+0000 7fb7b6788700 -1 osd.11 0 waiting for initial osdmap 2023-07-14T19:22:55.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:22:55 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:22:55.009+0000 7fb7b0d7b700 -1 osd.11 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:22:55.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:22:55.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T19:22:55.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:55.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='osd.11 [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:22:55.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:22:55.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2940104625' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:22:55.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:55.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:55.263 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:54 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:22:55.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T19:22:55.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:55.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='osd.11 [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:22:55.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:22:55.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2940104625' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:22:55.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:55.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:55.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:54 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:22:55.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T19:22:55.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:55.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='osd.11 [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:22:55.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2023-07-14T19:22:55.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2940104625' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:22:55.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:22:55.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:55.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:55.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:22:55.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:54 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:22:56.100 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T19:22:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:55 smithi104 ceph-mon[98435]: Detected new or changed devices on smithi104 2023-07-14T19:22:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:55 smithi104 ceph-mon[98435]: Adjusting osd_memory_target on smithi104 to 4292M 2023-07-14T19:22:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:55 smithi104 ceph-mon[98435]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:22:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:55 smithi104 ceph-mon[98435]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T19:22:56.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:55 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:56.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:55 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:56.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:55 smithi104 ceph-mon[98435]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:56.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:55 smithi053 ceph-mon[96565]: Detected new or changed devices on smithi104 2023-07-14T19:22:56.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:55 smithi053 ceph-mon[96565]: Adjusting osd_memory_target on smithi104 to 4292M 2023-07-14T19:22:56.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:55 smithi053 ceph-mon[96565]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:22:56.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:55 smithi053 ceph-mon[96565]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T19:22:56.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:55 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:56.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:55 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:56.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:55 smithi053 ceph-mon[96565]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:56.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:55 smithi070 ceph-mon[98015]: Detected new or changed devices on smithi104 2023-07-14T19:22:56.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:55 smithi070 ceph-mon[98015]: Adjusting osd_memory_target on smithi104 to 4292M 2023-07-14T19:22:56.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:55 smithi070 ceph-mon[98015]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2023-07-14T19:22:56.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:55 smithi070 ceph-mon[98015]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T19:22:56.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:55 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:56.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:55 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:56.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:55 smithi070 ceph-mon[98015]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:22:56.678 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:22:57.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:57 smithi104 ceph-mon[98435]: purged_snaps scrub starts 2023-07-14T19:22:57.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:57 smithi104 ceph-mon[98435]: purged_snaps scrub ok 2023-07-14T19:22:57.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:57 smithi104 ceph-mon[98435]: osd.11 [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] boot 2023-07-14T19:22:57.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:57 smithi104 ceph-mon[98435]: osdmap e75: 12 total, 12 up, 12 in 2023-07-14T19:22:57.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:57 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:57.300 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:57 smithi053 ceph-mon[96565]: purged_snaps scrub starts 2023-07-14T19:22:57.300 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:57 smithi053 ceph-mon[96565]: purged_snaps scrub ok 2023-07-14T19:22:57.301 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:57 smithi053 ceph-mon[96565]: osd.11 [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] boot 2023-07-14T19:22:57.301 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:57 smithi053 ceph-mon[96565]: osdmap e75: 12 total, 12 up, 12 in 2023-07-14T19:22:57.301 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:57 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:57.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:57 smithi070 ceph-mon[98015]: purged_snaps scrub starts 2023-07-14T19:22:57.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:57 smithi070 ceph-mon[98015]: purged_snaps scrub ok 2023-07-14T19:22:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:57 smithi070 ceph-mon[98015]: osd.11 [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] boot 2023-07-14T19:22:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:57 smithi070 ceph-mon[98015]: osdmap e75: 12 total, 12 up, 12 in 2023-07-14T19:22:57.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:57 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:22:57.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.987+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2407970505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41030c0 msgr2=0x7fdfb41054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:57.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.987+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2407970505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41030c0 0x7fdfb41054a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009990 tx=0x7fdfa4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:22:57.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.987+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2407970505 shutdown_connections 2023-07-14T19:22:57.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.987+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2407970505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfb41059e0 0x7fdfb4107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:57.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.987+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2407970505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfb41007a0 0x7fdfb4102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:57.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.987+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2407970505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41030c0 0x7fdfb41054a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:57.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.987+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2407970505 >> 172.21.15.53:0/2407970505 conn(0x7fdfb40fa1e0 msgr2=0x7fdfb40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:57.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.987+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2407970505 shutdown_connections 2023-07-14T19:22:57.993 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.987+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2407970505 wait complete. 2023-07-14T19:22:57.993 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfbbccb700 1 Processor -- start 2023-07-14T19:22:57.993 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfbbccb700 1 -- start start 2023-07-14T19:22:57.993 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfbbccb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41007a0 0x7fdfb4100100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:57.993 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfbbccb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfb41030c0 0x7fdfb40fe750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:57.994 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfbbccb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfb41059e0 0x7fdfb40fec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:57.994 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfbbccb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40ff410 con 0x7fdfb41007a0 2023-07-14T19:22:57.994 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfbbccb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40ff580 con 0x7fdfb41030c0 2023-07-14T19:22:57.994 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfbbccb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb419fa50 con 0x7fdfb41059e0 2023-07-14T19:22:57.994 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9a67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41007a0 0x7fdfb4100100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:57.994 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9a67700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41007a0 0x7fdfb4100100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49836/0 (socket says 172.21.15.53:49836) 2023-07-14T19:22:57.994 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9a67700 1 -- 172.21.15.53:0/2099003380 learned_addr learned my addr 172.21.15.53:0/2099003380 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:22:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfba268700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfb41059e0 0x7fdfb40fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9266700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfb41030c0 0x7fdfb40fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9a67700 1 -- 172.21.15.53:0/2099003380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfb41059e0 msgr2=0x7fdfb40fec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9a67700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfb41059e0 0x7fdfb40fec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9a67700 1 -- 172.21.15.53:0/2099003380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfb41030c0 msgr2=0x7fdfb40fe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9a67700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfb41030c0 0x7fdfb40fe750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9a67700 1 -- 172.21.15.53:0/2099003380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa40095f0 con 0x7fdfb41007a0 2023-07-14T19:22:57.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfba268700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfb41059e0 0x7fdfb40fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:22:57.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.988+0000 7fdfb9266700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfb41030c0 0x7fdfb40fe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:22:57.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.989+0000 7fdfb9a67700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41007a0 0x7fdfb4100100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000d980 tx=0x7fdfb000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:57.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.989+0000 7fdfaaffd700 1 -- 172.21.15.53:0/2099003380 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb000bb60 con 0x7fdfb41007a0 2023-07-14T19:22:57.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.989+0000 7fdfaaffd700 1 -- 172.21.15.53:0/2099003380 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfb0007640 con 0x7fdfb41007a0 2023-07-14T19:22:57.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.989+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb419fd30 con 0x7fdfb41007a0 2023-07-14T19:22:57.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.989+0000 7fdfaaffd700 1 -- 172.21.15.53:0/2099003380 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb0004e90 con 0x7fdfb41007a0 2023-07-14T19:22:57.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.989+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb41a0180 con 0x7fdfb41007a0 2023-07-14T19:22:57.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.990+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb40fbd50 con 0x7fdfb41007a0 2023-07-14T19:22:57.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.991+0000 7fdfaaffd700 1 -- 172.21.15.53:0/2099003380 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fdfb000bcc0 con 0x7fdfb41007a0 2023-07-14T19:22:57.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.991+0000 7fdfaaffd700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfa00a1150 0x7fdfa00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:57.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.992+0000 7fdfb9266700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfa00a1150 0x7fdfa00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:57.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.992+0000 7fdfaaffd700 1 -- 172.21.15.53:0/2099003380 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdfb00cc8b0 con 0x7fdfb41007a0 2023-07-14T19:22:57.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.992+0000 7fdfb9266700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfa00a1150 0x7fdfa00a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400b5c0 tx=0x7fdfa4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:57.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:57.995+0000 7fdfaaffd700 1 -- 172.21.15.53:0/2099003380 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdfb00964a0 con 0x7fdfb41007a0 2023-07-14T19:22:58.195 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.194+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fdfb410b510 con 0x7fdfb41007a0 2023-07-14T19:22:58.196 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.194+0000 7fdfaaffd700 1 -- 172.21.15.53:0/2099003380 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7fdfb00160a0 con 0x7fdfb41007a0 2023-07-14T19:22:58.196 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:22:58.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfa00a1150 msgr2=0x7fdfa00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:58.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfa00a1150 0x7fdfa00a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400b5c0 tx=0x7fdfa4005fb0 comp rx=0 tx=0).stop 2023-07-14T19:22:58.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41007a0 msgr2=0x7fdfb4100100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:58.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41007a0 0x7fdfb4100100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000d980 tx=0x7fdfb000dd40 comp rx=0 tx=0).stop 2023-07-14T19:22:58.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 shutdown_connections 2023-07-14T19:22:58.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfa00a1150 0x7fdfa00a3600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:58.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfb41059e0 0x7fdfb40fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:58.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb41007a0 0x7fdfb4100100 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:58.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 --2- 172.21.15.53:0/2099003380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfb41030c0 0x7fdfb40fe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:58.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.196+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 >> 172.21.15.53:0/2099003380 conn(0x7fdfb40fa1e0 msgr2=0x7fdfb41014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:58.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.197+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 shutdown_connections 2023-07-14T19:22:58.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:22:58.197+0000 7fdfbbccb700 1 -- 172.21.15.53:0/2099003380 wait complete. 2023-07-14T19:22:58.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:58 smithi104 ceph-mon[98435]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T19:22:58.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:58 smithi104 ceph-mon[98435]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:22:58.327 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:58 smithi053 ceph-mon[96565]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T19:22:58.327 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:58 smithi053 ceph-mon[96565]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:22:58.477 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:58 smithi070 ceph-mon[98015]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T19:22:58.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:58 smithi070 ceph-mon[98015]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:22:58.636 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":76,"num_osds":12,"num_up_osds":12,"osd_up_since":1689362575,"num_in_osds":12,"osd_in_since":1689362558,"num_remapped_pgs":0} 2023-07-14T19:22:58.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:22:59.198 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:22:59.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:22:59 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2099003380' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:22:59.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:22:59 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2099003380' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:22:59.477 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:22:59 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2099003380' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:23:00.264 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:00 smithi053 ceph-mon[96565]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:00 smithi070 ceph-mon[98015]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:00.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:00 smithi104 ceph-mon[98435]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:00.523 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.521+0000 7ff9efe4a700 1 -- 172.21.15.53:0/594761722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e81008d0 msgr2=0x7ff9e81048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:00.523 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.521+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/594761722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e81008d0 0x7ff9e81048a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80099c0 tx=0x7ff9d8009cd0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.524 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.522+0000 7ff9efe4a700 1 -- 172.21.15.53:0/594761722 shutdown_connections 2023-07-14T19:23:00.524 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.522+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/594761722 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9e8104fd0 0x7ff9e810cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.524 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.522+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/594761722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9e80fff30 0x7ff9e8100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.524 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.522+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/594761722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e81008d0 0x7ff9e81048a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.524 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.522+0000 7ff9efe4a700 1 -- 172.21.15.53:0/594761722 >> 172.21.15.53:0/594761722 conn(0x7ff9e80fb790 msgr2=0x7ff9e80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:00.524 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.522+0000 7ff9efe4a700 1 -- 172.21.15.53:0/594761722 shutdown_connections 2023-07-14T19:23:00.525 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.522+0000 7ff9efe4a700 1 -- 172.21.15.53:0/594761722 wait complete. 2023-07-14T19:23:00.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.522+0000 7ff9efe4a700 1 Processor -- start 2023-07-14T19:23:00.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9efe4a700 1 -- start start 2023-07-14T19:23:00.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9efe4a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9e80fff30 0x7ff9e8195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:00.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9efe4a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9e81008d0 0x7ff9e8196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:00.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9efe4a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e8104fd0 0x7ff9e819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:00.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9efe4a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e819ad40 con 0x7ff9e8104fd0 2023-07-14T19:23:00.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9efe4a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e819aeb0 con 0x7ff9e81008d0 2023-07-14T19:23:00.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9efe4a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e819b070 con 0x7ff9e80fff30 2023-07-14T19:23:00.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9ee3e7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e8104fd0 0x7ff9e819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:00.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9ee3e7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e8104fd0 0x7ff9e819a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49854/0 (socket says 172.21.15.53:49854) 2023-07-14T19:23:00.528 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9ee3e7700 1 -- 172.21.15.53:0/4047165737 learned_addr learned my addr 172.21.15.53:0/4047165737 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:00.528 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9edbe6700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9e80fff30 0x7ff9e8195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:00.528 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9ed3e5700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9e81008d0 0x7ff9e8196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:00.528 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9ee3e7700 1 -- 172.21.15.53:0/4047165737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9e80fff30 msgr2=0x7ff9e8195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:00.528 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9ee3e7700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9e80fff30 0x7ff9e8195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.528 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9ee3e7700 1 -- 172.21.15.53:0/4047165737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9e81008d0 msgr2=0x7ff9e8196360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:00.528 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9ee3e7700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9e81008d0 0x7ff9e8196360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.529 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9ee3e7700 1 -- 172.21.15.53:0/4047165737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80095f0 con 0x7ff9e8104fd0 2023-07-14T19:23:00.529 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.523+0000 7ff9edbe6700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9e80fff30 0x7ff9e8195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:23:00.529 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.524+0000 7ff9ee3e7700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e8104fd0 0x7ff9e819a730 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0007ee0 tx=0x7ff9e000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:00.529 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.524+0000 7ff9deffd700 1 -- 172.21.15.53:0/4047165737 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0017070 con 0x7ff9e8104fd0 2023-07-14T19:23:00.529 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.524+0000 7ff9deffd700 1 -- 172.21.15.53:0/4047165737 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9e000f040 con 0x7ff9e8104fd0 2023-07-14T19:23:00.529 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.524+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e819b350 con 0x7ff9e8104fd0 2023-07-14T19:23:00.529 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.524+0000 7ff9deffd700 1 -- 172.21.15.53:0/4047165737 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0005740 con 0x7ff9e8104fd0 2023-07-14T19:23:00.529 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.524+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e81a2bb0 con 0x7ff9e8104fd0 2023-07-14T19:23:00.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.525+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e804e340 con 0x7ff9e8104fd0 2023-07-14T19:23:00.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.526+0000 7ff9deffd700 1 -- 172.21.15.53:0/4047165737 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff9e00058a0 con 0x7ff9e8104fd0 2023-07-14T19:23:00.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.527+0000 7ff9deffd700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff9d40a11d0 0x7ff9d40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:00.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.527+0000 7ff9edbe6700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff9d40a11d0 0x7ff9d40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:00.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.527+0000 7ff9deffd700 1 -- 172.21.15.53:0/4047165737 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff9e00cbca0 con 0x7ff9e8104fd0 2023-07-14T19:23:00.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.527+0000 7ff9edbe6700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff9d40a11d0 0x7ff9d40a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005950 tx=0x7ff9e4005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:00.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.531+0000 7ff9deffd700 1 -- 172.21.15.53:0/4047165737 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff9e00957f0 con 0x7ff9e8104fd0 2023-07-14T19:23:00.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.729+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9e81a2de0 con 0x7ff9e8104fd0 2023-07-14T19:23:00.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.730+0000 7ff9deffd700 1 -- 172.21.15.53:0/4047165737 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18357 (secure 0 0 0) 0x7ff9e0098e10 con 0x7ff9e8104fd0 2023-07-14T19:23:00.732 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:00.732 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":76,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:22:56.998864+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:23:00.734 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff9d40a11d0 msgr2=0x7ff9d40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:00.734 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff9d40a11d0 0x7ff9d40a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005950 tx=0x7ff9e4005f50 comp rx=0 tx=0).stop 2023-07-14T19:23:00.734 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e8104fd0 msgr2=0x7ff9e819a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:00.734 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e8104fd0 0x7ff9e819a730 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0007ee0 tx=0x7ff9e000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.734 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 shutdown_connections 2023-07-14T19:23:00.734 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff9d40a11d0 0x7ff9d40a3680 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9e8104fd0 0x7ff9e819a730 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9e80fff30 0x7ff9e8195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 --2- 172.21.15.53:0/4047165737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9e81008d0 0x7ff9e8196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.732+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 >> 172.21.15.53:0/4047165737 conn(0x7ff9e80fb790 msgr2=0x7ff9e80fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.733+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 shutdown_connections 2023-07-14T19:23:00.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:00.733+0000 7ff9efe4a700 1 -- 172.21.15.53:0/4047165737 wait complete. 2023-07-14T19:23:01.247 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:19:51.775369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:23:01.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-14T19:23:01.276 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:01 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/4047165737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:01.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:01 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/4047165737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:01.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:01 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/4047165737' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:01.808 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:02.299 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:02 smithi053 ceph-mon[96565]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:02.477 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:02 smithi070 ceph-mon[98015]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:02.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:02 smithi104 ceph-mon[98435]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:03.016 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.014+0000 7fce64fd1700 1 -- 172.21.15.53:0/872127212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce600fff60 msgr2=0x7fce60100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:03.016 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.014+0000 7fce64fd1700 1 --2- 172.21.15.53:0/872127212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce600fff60 0x7fce60100330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fce540099c0 tx=0x7fce54009cd0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.016 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.014+0000 7fce64fd1700 1 -- 172.21.15.53:0/872127212 shutdown_connections 2023-07-14T19:23:03.016 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.014+0000 7fce64fd1700 1 --2- 172.21.15.53:0/872127212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce60104ff0 0x7fce6010cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.017 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.014+0000 7fce64fd1700 1 --2- 172.21.15.53:0/872127212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce600fff60 0x7fce60100330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.017 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.014+0000 7fce64fd1700 1 --2- 172.21.15.53:0/872127212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce60100900 0x7fce601048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.017 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.014+0000 7fce64fd1700 1 -- 172.21.15.53:0/872127212 >> 172.21.15.53:0/872127212 conn(0x7fce60077ca0 msgr2=0x7fce600780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:03.017 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.014+0000 7fce64fd1700 1 -- 172.21.15.53:0/872127212 shutdown_connections 2023-07-14T19:23:03.017 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.014+0000 7fce64fd1700 1 -- 172.21.15.53:0/872127212 wait complete. 2023-07-14T19:23:03.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce64fd1700 1 Processor -- start 2023-07-14T19:23:03.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce64fd1700 1 -- start start 2023-07-14T19:23:03.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce64fd1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce600fff60 0x7fce60195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:03.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce64fd1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce60100900 0x7fce60196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:03.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce64fd1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce60104ff0 0x7fce6019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:03.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce64fd1700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6019ad80 con 0x7fce60104ff0 2023-07-14T19:23:03.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce64fd1700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6019aef0 con 0x7fce60100900 2023-07-14T19:23:03.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce64fd1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6019b0b0 con 0x7fce600fff60 2023-07-14T19:23:03.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce5ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce60104ff0 0x7fce6019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:03.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce5ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce60104ff0 0x7fce6019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49868/0 (socket says 172.21.15.53:49868) 2023-07-14T19:23:03.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.015+0000 7fce5ed9d700 1 -- 172.21.15.53:0/3616869734 learned_addr learned my addr 172.21.15.53:0/3616869734 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:03.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce5e59c700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce600fff60 0x7fce60195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:03.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce5dd9b700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce60100900 0x7fce60196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:03.021 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce5ed9d700 1 -- 172.21.15.53:0/3616869734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce600fff60 msgr2=0x7fce60195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:03.021 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce5ed9d700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce600fff60 0x7fce60195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.021 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce5ed9d700 1 -- 172.21.15.53:0/3616869734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce60100900 msgr2=0x7fce60196300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:03.021 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce5ed9d700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce60100900 0x7fce60196300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.021 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce5ed9d700 1 -- 172.21.15.53:0/3616869734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce540095f0 con 0x7fce60104ff0 2023-07-14T19:23:03.021 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce5ed9d700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce60104ff0 0x7fce6019a770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fce58007ee0 tx=0x7fce5800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:03.022 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce4f7fe700 1 -- 172.21.15.53:0/3616869734 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5800dd20 con 0x7fce60104ff0 2023-07-14T19:23:03.022 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6019b390 con 0x7fce60104ff0 2023-07-14T19:23:03.022 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce4f7fe700 1 -- 172.21.15.53:0/3616869734 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce5800f040 con 0x7fce60104ff0 2023-07-14T19:23:03.022 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce4f7fe700 1 -- 172.21.15.53:0/3616869734 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce580048a0 con 0x7fce60104ff0 2023-07-14T19:23:03.022 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.016+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce601a2bf0 con 0x7fce60104ff0 2023-07-14T19:23:03.022 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.017+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6004e340 con 0x7fce60104ff0 2023-07-14T19:23:03.023 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.018+0000 7fce4f7fe700 1 -- 172.21.15.53:0/3616869734 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fce58003a00 con 0x7fce60104ff0 2023-07-14T19:23:03.023 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.019+0000 7fce4f7fe700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fce480a1150 0x7fce480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:03.024 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.019+0000 7fce5e59c700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fce480a1150 0x7fce480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:03.024 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.019+0000 7fce4f7fe700 1 -- 172.21.15.53:0/3616869734 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fce580cc4d0 con 0x7fce60104ff0 2023-07-14T19:23:03.024 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.019+0000 7fce5e59c700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fce480a1150 0x7fce480a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fce54000c00 tx=0x7fce54003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:03.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.022+0000 7fce4f7fe700 1 -- 172.21.15.53:0/3616869734 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce58096020 con 0x7fce60104ff0 2023-07-14T19:23:03.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.218+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fce601a2d80 con 0x7fce60104ff0 2023-07-14T19:23:03.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.219+0000 7fce4f7fe700 1 -- 172.21.15.53:0/3616869734 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fce58099640 con 0x7fce60104ff0 2023-07-14T19:23:03.220 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2023-07-14T19:23:03.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fce480a1150 msgr2=0x7fce480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fce480a1150 0x7fce480a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fce54000c00 tx=0x7fce54003680 comp rx=0 tx=0).stop 2023-07-14T19:23:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce60104ff0 msgr2=0x7fce6019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce60104ff0 0x7fce6019a770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fce58007ee0 tx=0x7fce5800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 shutdown_connections 2023-07-14T19:23:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fce480a1150 0x7fce480a3600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce60104ff0 0x7fce6019a770 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fce600fff60 0x7fce60195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 --2- 172.21.15.53:0/3616869734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce60100900 0x7fce60196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 >> 172.21.15.53:0/3616869734 conn(0x7fce60077ca0 msgr2=0x7fce601038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:03.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 shutdown_connections 2023-07-14T19:23:03.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:03.221+0000 7fce64fd1700 1 -- 172.21.15.53:0/3616869734 wait complete. 2023-07-14T19:23:03.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:03 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3616869734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:23:03.668 INFO:tasks.cephadm:Adding mds.a on smithi053 2023-07-14T19:23:03.668 INFO:tasks.cephadm:Adding mds.d on smithi053 2023-07-14T19:23:03.669 INFO:tasks.cephadm:Adding mds.g on smithi053 2023-07-14T19:23:03.669 INFO:tasks.cephadm:Adding mds.j on smithi053 2023-07-14T19:23:03.670 INFO:tasks.cephadm:Adding mds.b on smithi070 2023-07-14T19:23:03.670 INFO:tasks.cephadm:Adding mds.e on smithi070 2023-07-14T19:23:03.670 INFO:tasks.cephadm:Adding mds.h on smithi070 2023-07-14T19:23:03.671 INFO:tasks.cephadm:Adding mds.k on smithi070 2023-07-14T19:23:03.671 INFO:tasks.cephadm:Adding mds.c on smithi104 2023-07-14T19:23:03.671 INFO:tasks.cephadm:Adding mds.f on smithi104 2023-07-14T19:23:03.672 INFO:tasks.cephadm:Adding mds.i on smithi104 2023-07-14T19:23:03.672 INFO:tasks.cephadm:Adding mds.l on smithi104 2023-07-14T19:23:03.672 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph orch apply mds all '12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l' 2023-07-14T19:23:03.705 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:03 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3616869734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:23:03.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:03 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3616869734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:23:04.238 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:23:04.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:04 smithi053 ceph-mon[96565]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:04.586 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:04 smithi104 ceph-mon[98435]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:04.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:04 smithi070 ceph-mon[98015]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2219097580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0fe340 msgr2=0x7fb9dc0fe750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2219097580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0fe340 0x7fb9dc0fe750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009990 tx=0x7fb9d0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2219097580 shutdown_connections 2023-07-14T19:23:05.406 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2219097580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc0ffed0 0x7fb9dc106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2219097580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0fe340 0x7fb9dc0fe750 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2219097580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9dc0ff540 0x7fb9dc0ff990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2219097580 >> 172.21.15.104:0/2219097580 conn(0x7fb9dc0f9990 msgr2=0x7fb9dc0fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2219097580 shutdown_connections 2023-07-14T19:23:05.407 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2219097580 wait complete. 2023-07-14T19:23:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 Processor -- start 2023-07-14T19:23:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.404+0000 7fb9e2a87700 1 -- start start 2023-07-14T19:23:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9e2a87700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc0fe340 0x7fb9dc1022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9e2a87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0ff540 0x7fb9dc100920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:05.408 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9e2a87700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9dc0ffed0 0x7fb9dc100e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9e2a87700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc1015e0 con 0x7fb9dc0ffed0 2023-07-14T19:23:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9e2a87700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc101750 con 0x7fb9dc0fe340 2023-07-14T19:23:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9e2a87700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc19fb20 con 0x7fb9dc0ff540 2023-07-14T19:23:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9dbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0ff540 0x7fb9dc100920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9dbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0ff540 0x7fb9dc100920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:57372/0 (socket says 172.21.15.104:57372) 2023-07-14T19:23:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9dbfff700 1 -- 172.21.15.104:0/2876863611 learned_addr learned my addr 172.21.15.104:0/2876863611 (peer_addr_for_me v2:172.21.15.104:0/0) 2023-07-14T19:23:05.409 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9e0823700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc0fe340 0x7fb9dc1022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:05.410 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9e1024700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9dc0ffed0 0x7fb9dc100e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:05.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9dbfff700 1 -- 172.21.15.104:0/2876863611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc0fe340 msgr2=0x7fb9dc1022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:05.411 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9dbfff700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc0fe340 0x7fb9dc1022d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9dbfff700 1 -- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9dc0ffed0 msgr2=0x7fb9dc100e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9dbfff700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9dc0ffed0 0x7fb9dc100e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9dbfff700 1 -- 172.21.15.104:0/2876863611 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d00095f0 con 0x7fb9dc0ff540 2023-07-14T19:23:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9dbfff700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0ff540 0x7fb9dc100920 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800b530 tx=0x7fb9c800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9d9ffb700 1 -- 172.21.15.104:0/2876863611 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8011830 con 0x7fb9dc0ff540 2023-07-14T19:23:05.412 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9d9ffb700 1 -- 172.21.15.104:0/2876863611 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9c8011e50 con 0x7fb9dc0ff540 2023-07-14T19:23:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc19fe00 con 0x7fb9dc0ff540 2023-07-14T19:23:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.405+0000 7fb9d9ffb700 1 -- 172.21.15.104:0/2876863611 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c800f2e0 con 0x7fb9dc0ff540 2023-07-14T19:23:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.406+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc1a0270 con 0x7fb9dc0ff540 2023-07-14T19:23:05.413 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.407+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc109c40 con 0x7fb9dc0ff540 2023-07-14T19:23:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.407+0000 7fb9d9ffb700 1 -- 172.21.15.104:0/2876863611 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb9c80105a0 con 0x7fb9dc0ff540 2023-07-14T19:23:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.408+0000 7fb9d9ffb700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb9cc0a1150 0x7fb9cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.408+0000 7fb9d9ffb700 1 -- 172.21.15.104:0/2876863611 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb9c80cbdf0 con 0x7fb9dc0ff540 2023-07-14T19:23:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.408+0000 7fb9e0823700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb9cc0a1150 0x7fb9cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:05.414 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.409+0000 7fb9e0823700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb9cc0a1150 0x7fb9cc0a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000b5c0 tx=0x7fb9d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:05.415 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.413+0000 7fb9d9ffb700 1 -- 172.21.15.104:0/2876863611 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb9c80959e0 con 0x7fb9dc0ff540 2023-07-14T19:23:05.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:05 smithi104 ceph-mon[98435]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:05.610 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.608+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9dc105a90 con 0x7fb9cc0a1150 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.617+0000 7fb9d9ffb700 1 -- 172.21.15.104:0/2876863611 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb9dc105a90 con 0x7fb9cc0a1150 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mds.all update... 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb9cc0a1150 msgr2=0x7fb9cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb9cc0a1150 0x7fb9cc0a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000b5c0 tx=0x7fb9d0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0ff540 msgr2=0x7fb9dc100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0ff540 0x7fb9dc100920 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800b530 tx=0x7fb9c800b840 comp rx=0 tx=0).stop 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 shutdown_connections 2023-07-14T19:23:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb9cc0a1150 0x7fb9cc0a3600 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9dc0ffed0 0x7fb9dc100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9dc0fe340 0x7fb9dc1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 --2- 172.21.15.104:0/2876863611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9dc0ff540 0x7fb9dc100920 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 >> 172.21.15.104:0/2876863611 conn(0x7fb9dc0f9990 msgr2=0x7fb9dc0fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 shutdown_connections 2023-07-14T19:23:05.622 INFO:teuthology.orchestra.run.smithi104.stderr:2023-07-14T19:23:05.620+0000 7fb9e2a87700 1 -- 172.21.15.104:0/2876863611 wait complete. 2023-07-14T19:23:05.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:05 smithi070 ceph-mon[98015]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:05.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:05 smithi053 ceph-mon[96565]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:05.979 DEBUG:teuthology.orchestra.run.smithi053:mds.a> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.a.service 2023-07-14T19:23:05.984 DEBUG:teuthology.orchestra.run.smithi053:mds.d> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.d.service 2023-07-14T19:23:06.022 INFO:journalctl@ceph.mds.a.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:23:06.024 DEBUG:teuthology.orchestra.run.smithi053:mds.g> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.g.service 2023-07-14T19:23:06.053 INFO:journalctl@ceph.mds.d.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:23:06.054 DEBUG:teuthology.orchestra.run.smithi053:mds.j> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.j.service 2023-07-14T19:23:06.083 INFO:journalctl@ceph.mds.g.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:23:06.084 DEBUG:teuthology.orchestra.run.smithi070:mds.b> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.b.service 2023-07-14T19:23:06.087 DEBUG:teuthology.orchestra.run.smithi070:mds.e> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.e.service 2023-07-14T19:23:06.111 INFO:journalctl@ceph.mds.j.smithi053.stdout:-- Logs begin at Fri 2023-07-14 18:52:43 UTC. -- 2023-07-14T19:23:06.124 INFO:journalctl@ceph.mds.b.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:23:06.125 DEBUG:teuthology.orchestra.run.smithi070:mds.h> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.h.service 2023-07-14T19:23:06.154 INFO:journalctl@ceph.mds.e.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:23:06.156 DEBUG:teuthology.orchestra.run.smithi070:mds.k> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.k.service 2023-07-14T19:23:06.182 INFO:journalctl@ceph.mds.h.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:23:06.183 DEBUG:teuthology.orchestra.run.smithi104:mds.c> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.c.service 2023-07-14T19:23:06.186 DEBUG:teuthology.orchestra.run.smithi104:mds.f> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.f.service 2023-07-14T19:23:06.209 INFO:journalctl@ceph.mds.k.smithi070.stdout:-- Logs begin at Fri 2023-07-14 18:53:12 UTC. -- 2023-07-14T19:23:06.215 INFO:journalctl@ceph.mds.c.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:23:06.216 DEBUG:teuthology.orchestra.run.smithi104:mds.i> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.i.service 2023-07-14T19:23:06.242 INFO:journalctl@ceph.mds.f.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:23:06.243 DEBUG:teuthology.orchestra.run.smithi104:mds.l> sudo journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.l.service 2023-07-14T19:23:06.268 INFO:journalctl@ceph.mds.i.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:23:06.269 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-14T19:23:06.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:23:06.296 INFO:journalctl@ceph.mds.l.smithi104.stdout:-- Logs begin at Fri 2023-07-14 18:53:17 UTC. -- 2023-07-14T19:23:06.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.418+0000 7fb3626d7700 1 -- 172.21.15.53:0/824117467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c100680 msgr2=0x7fb35c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.418+0000 7fb3626d7700 1 --2- 172.21.15.53:0/824117467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c100680 0x7fb35c106f40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb35400b0a0 tx=0x7fb35400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.418+0000 7fb3626d7700 1 -- 172.21.15.53:0/824117467 shutdown_connections 2023-07-14T19:23:06.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.418+0000 7fb3626d7700 1 --2- 172.21.15.53:0/824117467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c100680 0x7fb35c106f40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.418+0000 7fb3626d7700 1 --2- 172.21.15.53:0/824117467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c0feb00 0x7fb35c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.418+0000 7fb3626d7700 1 --2- 172.21.15.53:0/824117467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb35c0ffcf0 0x7fb35c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.418+0000 7fb3626d7700 1 -- 172.21.15.53:0/824117467 >> 172.21.15.53:0/824117467 conn(0x7fb35c0fa100 msgr2=0x7fb35c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:06.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.418+0000 7fb3626d7700 1 -- 172.21.15.53:0/824117467 shutdown_connections 2023-07-14T19:23:06.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.418+0000 7fb3626d7700 1 -- 172.21.15.53:0/824117467 wait complete. 2023-07-14T19:23:06.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb3626d7700 1 Processor -- start 2023-07-14T19:23:06.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb3626d7700 1 -- start start 2023-07-14T19:23:06.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb3626d7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c0feb00 0x7fb35c068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:06.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb3626d7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c0ffcf0 0x7fb35c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:06.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb3626d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb35c100680 0x7fb35c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:06.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb3626d7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c19d540 con 0x7fb35c0feb00 2023-07-14T19:23:06.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb3626d7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c19d6b0 con 0x7fb35c0ffcf0 2023-07-14T19:23:06.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb3626d7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c19d870 con 0x7fb35c100680 2023-07-14T19:23:06.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb35bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c0feb00 0x7fb35c068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:06.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb35bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c0feb00 0x7fb35c068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49876/0 (socket says 172.21.15.53:49876) 2023-07-14T19:23:06.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb35bfff700 1 -- 172.21.15.53:0/3056909355 learned_addr learned my addr 172.21.15.53:0/3056909355 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:06.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.419+0000 7fb360c74700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb35c100680 0x7fb35c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:06.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb35b7fe700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c0ffcf0 0x7fb35c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:06.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb35bfff700 1 -- 172.21.15.53:0/3056909355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb35c100680 msgr2=0x7fb35c19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb35bfff700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb35c100680 0x7fb35c19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb35bfff700 1 -- 172.21.15.53:0/3056909355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c0ffcf0 msgr2=0x7fb35c19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb35bfff700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c0ffcf0 0x7fb35c19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb35bfff700 1 -- 172.21.15.53:0/3056909355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb354009d00 con 0x7fb35c0feb00 2023-07-14T19:23:06.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb35bfff700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c0feb00 0x7fb35c068310 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00e970 tx=0x7fb34c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:06.430 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb3597fa700 1 -- 172.21.15.53:0/3056909355 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c009790 con 0x7fb35c0feb00 2023-07-14T19:23:06.431 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb3597fa700 1 -- 172.21.15.53:0/3056909355 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb34c00c7e0 con 0x7fb35c0feb00 2023-07-14T19:23:06.432 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb3597fa700 1 -- 172.21.15.53:0/3056909355 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c003c70 con 0x7fb35c0feb00 2023-07-14T19:23:06.432 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c19db50 con 0x7fb35c0feb00 2023-07-14T19:23:06.432 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.420+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c19e060 con 0x7fb35c0feb00 2023-07-14T19:23:06.433 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.421+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c105fd0 con 0x7fb35c0feb00 2023-07-14T19:23:06.433 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.422+0000 7fb3597fa700 1 -- 172.21.15.53:0/3056909355 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb34c00cc50 con 0x7fb35c0feb00 2023-07-14T19:23:06.433 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.423+0000 7fb3597fa700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3480a1220 0x7fb3480a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:06.433 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.423+0000 7fb3597fa700 1 -- 172.21.15.53:0/3056909355 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb34c014070 con 0x7fb35c0feb00 2023-07-14T19:23:06.434 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.423+0000 7fb35b7fe700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3480a1220 0x7fb3480a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:06.434 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.423+0000 7fb35b7fe700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3480a1220 0x7fb3480a36d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb344009c60 tx=0x7fb344009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:06.435 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.426+0000 7fb3597fa700 1 -- 172.21.15.53:0/3056909355 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb34c0962f0 con 0x7fb35c0feb00 2023-07-14T19:23:06.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.627+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb35c02c9d0 con 0x7fb35c0feb00 2023-07-14T19:23:06.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.628+0000 7fb3597fa700 1 -- 172.21.15.53:0/3056909355 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18357 (secure 0 0 0) 0x7fb34c003dd0 con 0x7fb35c0feb00 2023-07-14T19:23:06.631 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:06.631 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":76,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:22:56.998864+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:23:06.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3480a1220 msgr2=0x7fb3480a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3480a1220 0x7fb3480a36d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb344009c60 tx=0x7fb344009260 comp rx=0 tx=0).stop 2023-07-14T19:23:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c0feb00 msgr2=0x7fb35c068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c0feb00 0x7fb35c068310 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00e970 tx=0x7fb34c00ec80 comp rx=0 tx=0).stop 2023-07-14T19:23:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 shutdown_connections 2023-07-14T19:23:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3480a1220 0x7fb3480a36d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb35c100680 0x7fb35c19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb35c0feb00 0x7fb35c068310 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 --2- 172.21.15.53:0/3056909355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35c0ffcf0 0x7fb35c19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 >> 172.21.15.53:0/3056909355 conn(0x7fb35c0fa100 msgr2=0x7fb35c103110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:06.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 shutdown_connections 2023-07-14T19:23:06.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.630+0000 7fb3626d7700 1 -- 172.21.15.53:0/3056909355 wait complete. 2023-07-14T19:23:06.645 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:19:51.775369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:23:06.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:23:06.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.772+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3756584465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 msgr2=0x7fc0e4105430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.772+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3756584465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 0x7fc0e4105430 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009990 tx=0x7fc0d4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.772+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3756584465 shutdown_connections 2023-07-14T19:23:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.772+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3756584465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4105970 0x7fc0e4107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.772+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3756584465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e4100730 0x7fc0e4102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.772+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3756584465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 0x7fc0e4105430 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.772+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3756584465 >> 172.21.15.53:0/3756584465 conn(0x7fc0e40fa130 msgr2=0x7fc0e40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.772+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3756584465 shutdown_connections 2023-07-14T19:23:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.772+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3756584465 wait complete. 2023-07-14T19:23:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ec34a700 1 Processor -- start 2023-07-14T19:23:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ec34a700 1 -- start start 2023-07-14T19:23:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ec34a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4100730 0x7fc0e4195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ec34a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 0x7fc0e4195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ec34a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e4105970 0x7fc0e419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ec34a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e419d4f0 con 0x7fc0e4103050 2023-07-14T19:23:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ec34a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e419d630 con 0x7fc0e4105970 2023-07-14T19:23:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ec34a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e419d7c0 con 0x7fc0e4100730 2023-07-14T19:23:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0e98e5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 0x7fc0e4195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0e98e5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 0x7fc0e4195eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49892/0 (socket says 172.21.15.53:49892) 2023-07-14T19:23:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0e98e5700 1 -- 172.21.15.53:0/3688352522 learned_addr learned my addr 172.21.15.53:0/3688352522 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ea0e6700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4100730 0x7fc0e4195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0ea8e7700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e4105970 0x7fc0e419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0e98e5700 1 -- 172.21.15.53:0/3688352522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4100730 msgr2=0x7fc0e4195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0e98e5700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4100730 0x7fc0e4195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0e98e5700 1 -- 172.21.15.53:0/3688352522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e4105970 msgr2=0x7fc0e419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0e98e5700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e4105970 0x7fc0e419cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0e98e5700 1 -- 172.21.15.53:0/3688352522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0dc009c40 con 0x7fc0e4103050 2023-07-14T19:23:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.773+0000 7fc0e98e5700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 0x7fc0e4195eb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4004740 tx=0x7fc0d4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.774+0000 7fc0db7fe700 1 -- 172.21.15.53:0/3688352522 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d401d070 con 0x7fc0e4103050 2023-07-14T19:23:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.774+0000 7fc0db7fe700 1 -- 172.21.15.53:0/3688352522 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0d4004830 con 0x7fc0e4103050 2023-07-14T19:23:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.774+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d40095f0 con 0x7fc0e4103050 2023-07-14T19:23:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.774+0000 7fc0db7fe700 1 -- 172.21.15.53:0/3688352522 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40053c0 con 0x7fc0e4103050 2023-07-14T19:23:06.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.774+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e419dc70 con 0x7fc0e4103050 2023-07-14T19:23:06.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.774+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e40fbca0 con 0x7fc0e4103050 2023-07-14T19:23:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.776+0000 7fc0db7fe700 1 -- 172.21.15.53:0/3688352522 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc0d4005560 con 0x7fc0e4103050 2023-07-14T19:23:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.776+0000 7fc0db7fe700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc0d00a1150 0x7fc0d00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.776+0000 7fc0db7fe700 1 -- 172.21.15.53:0/3688352522 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc0d40cda50 con 0x7fc0e4103050 2023-07-14T19:23:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.776+0000 7fc0ea0e6700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc0d00a1150 0x7fc0d00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.777+0000 7fc0ea0e6700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc0d00a1150 0x7fc0d00a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009c60 tx=0x7fc0e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.780+0000 7fc0db7fe700 1 -- 172.21.15.53:0/3688352522 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc0d40976d0 con 0x7fc0e4103050 2023-07-14T19:23:06.957 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:06.957 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: Saving service mds.all spec with placement smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l;count:12 2023-07-14T19:23:06.958 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:06.958 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:06.958 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:06.958 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:06.958 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:06.958 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:06.958 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:06.959 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:06.959 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:06 smithi104 ceph-mon[98435]: Deploying daemon mds.l on smithi104 2023-07-14T19:23:06.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:06.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: Saving service mds.all spec with placement smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l;count:12 2023-07-14T19:23:06.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:06.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:06.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:06.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:06.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:06.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:06.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:06.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:06.980 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:06 smithi070 ceph-mon[98015]: Deploying daemon mds.l on smithi104 2023-07-14T19:23:06.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.979+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0e402ca70 con 0x7fc0e4103050 2023-07-14T19:23:06.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.979+0000 7fc0db7fe700 1 -- 172.21.15.53:0/3688352522 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0d409acf0 con 0x7fc0e4103050 2023-07-14T19:23:06.981 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2023-07-14T19:23:06.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc0d00a1150 msgr2=0x7fc0d00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc0d00a1150 0x7fc0d00a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009c60 tx=0x7fc0e0009260 comp rx=0 tx=0).stop 2023-07-14T19:23:06.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 msgr2=0x7fc0e4195eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 0x7fc0e4195eb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4004740 tx=0x7fc0d4009580 comp rx=0 tx=0).stop 2023-07-14T19:23:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 shutdown_connections 2023-07-14T19:23:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc0d00a1150 0x7fc0d00a3600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc0e4105970 0x7fc0e419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc0e4100730 0x7fc0e4195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 --2- 172.21.15.53:0/3688352522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e4103050 0x7fc0e4195eb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:06.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 >> 172.21.15.53:0/3688352522 conn(0x7fc0e40fa130 msgr2=0x7fc0e4101390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:06.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 shutdown_connections 2023-07-14T19:23:06.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:06.982+0000 7fc0ec34a700 1 -- 172.21.15.53:0/3688352522 wait complete. 2023-07-14T19:23:06.995 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:07.040 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: Saving service mds.all spec with placement smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l;count:12 2023-07-14T19:23:07.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:07.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:07.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:07.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:07.041 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:07.042 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:07.042 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:07.042 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:07.042 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:06 smithi053 ceph-mon[96565]: Deploying daemon mds.l on smithi104 2023-07-14T19:23:07.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:07.172 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.168+0000 7f55c07fe700 1 -- 172.21.15.53:0/1622626592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55b8100180 msgr2=0x7f55b8100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:07.172 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.168+0000 7f55c07fe700 1 --2- 172.21.15.53:0/1622626592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55b8100180 0x7f55b8100590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f55a80098f0 tx=0x7f55a8009c00 comp rx=0 tx=0).stop 2023-07-14T19:23:07.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.168+0000 7f55c07fe700 1 -- 172.21.15.53:0/1622626592 shutdown_connections 2023-07-14T19:23:07.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.168+0000 7f55c07fe700 1 --2- 172.21.15.53:0/1622626592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b80ff6f0 0x7f55b8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.168+0000 7f55c07fe700 1 --2- 172.21.15.53:0/1622626592 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55b8100180 0x7f55b8100590 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.168+0000 7f55c07fe700 1 --2- 172.21.15.53:0/1622626592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55b80fed40 0x7f55b80ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.168+0000 7f55c07fe700 1 -- 172.21.15.53:0/1622626592 >> 172.21.15.53:0/1622626592 conn(0x7f55b80f98a0 msgr2=0x7f55b80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:07.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.168+0000 7f55c07fe700 1 -- 172.21.15.53:0/1622626592 shutdown_connections 2023-07-14T19:23:07.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.168+0000 7f55c07fe700 1 -- 172.21.15.53:0/1622626592 wait complete. 2023-07-14T19:23:07.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55c07fe700 1 Processor -- start 2023-07-14T19:23:07.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55c07fe700 1 -- start start 2023-07-14T19:23:07.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55c07fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55b80fed40 0x7f55b8197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:07.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55c07fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b80ff6f0 0x7f55b81980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:07.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55c07fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55b8100180 0x7f55b819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:07.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55c07fe700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b819f720 con 0x7f55b80ff6f0 2023-07-14T19:23:07.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55c07fe700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b819f860 con 0x7f55b80fed40 2023-07-14T19:23:07.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55c07fe700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b819f9f0 con 0x7f55b8100180 2023-07-14T19:23:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bdd99700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b80ff6f0 0x7f55b81980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bdd99700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b80ff6f0 0x7f55b81980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49906/0 (socket says 172.21.15.53:49906) 2023-07-14T19:23:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bdd99700 1 -- 172.21.15.53:0/2351029457 learned_addr learned my addr 172.21.15.53:0/2351029457 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55be59a700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55b80fed40 0x7f55b8197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bed9b700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55b8100180 0x7f55b819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bdd99700 1 -- 172.21.15.53:0/2351029457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55b8100180 msgr2=0x7f55b819f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bdd99700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55b8100180 0x7f55b819f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bdd99700 1 -- 172.21.15.53:0/2351029457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55b80fed40 msgr2=0x7f55b8197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bdd99700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55b80fed40 0x7f55b8197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bdd99700 1 -- 172.21.15.53:0/2351029457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b0009d00 con 0x7f55b80ff6f0 2023-07-14T19:23:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.169+0000 7f55bdd99700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b80ff6f0 0x7f55b81980e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55b400e9a0 tx=0x7f55b400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.170+0000 7f55af7fe700 1 -- 172.21.15.53:0/2351029457 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b4009790 con 0x7f55b80ff6f0 2023-07-14T19:23:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.170+0000 7f55af7fe700 1 -- 172.21.15.53:0/2351029457 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f55b4010430 con 0x7f55b80ff6f0 2023-07-14T19:23:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.170+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a8009520 con 0x7f55b80ff6f0 2023-07-14T19:23:07.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.170+0000 7f55af7fe700 1 -- 172.21.15.53:0/2351029457 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b4018680 con 0x7f55b80ff6f0 2023-07-14T19:23:07.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.170+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b81a00a0 con 0x7f55b80ff6f0 2023-07-14T19:23:07.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.171+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b810a320 con 0x7f55b80ff6f0 2023-07-14T19:23:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.172+0000 7f55af7fe700 1 -- 172.21.15.53:0/2351029457 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f55b40105a0 con 0x7f55b80ff6f0 2023-07-14T19:23:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.173+0000 7f55af7fe700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f55a40a1220 0x7f55a40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.173+0000 7f55af7fe700 1 -- 172.21.15.53:0/2351029457 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f55b4014070 con 0x7f55b80ff6f0 2023-07-14T19:23:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.173+0000 7f55be59a700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f55a40a1220 0x7f55a40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.173+0000 7f55be59a700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f55a40a1220 0x7f55a40a36d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55a8005e90 tx=0x7f55a8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.176+0000 7f55af7fe700 1 -- 172.21.15.53:0/2351029457 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55b40963c0 con 0x7f55b80ff6f0 2023-07-14T19:23:07.260 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:23:07 smithi104 systemd[1]: Starting Ceph mds.l for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:07.423 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.421+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55b810a510 con 0x7f55b80ff6f0 2023-07-14T19:23:07.423 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.422+0000 7f55af7fe700 1 -- 172.21.15.53:0/2351029457 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f55b4026210 con 0x7f55b80ff6f0 2023-07-14T19:23:07.424 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:07.424 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-14T19:23:07.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f55a40a1220 msgr2=0x7f55a40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:07.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f55a40a1220 0x7f55a40a36d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55a8005e90 tx=0x7f55a8019040 comp rx=0 tx=0).stop 2023-07-14T19:23:07.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b80ff6f0 msgr2=0x7f55b81980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:07.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b80ff6f0 0x7f55b81980e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f55b400e9a0 tx=0x7f55b400ecb0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 shutdown_connections 2023-07-14T19:23:07.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f55a40a1220 0x7f55a40a36d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55b8100180 0x7f55b819f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55b80fed40 0x7f55b8197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 --2- 172.21.15.53:0/2351029457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b80ff6f0 0x7f55b81980e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 >> 172.21.15.53:0/2351029457 conn(0x7f55b80f98a0 msgr2=0x7f55b80fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:07.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 shutdown_connections 2023-07-14T19:23:07.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:07.424+0000 7f55c07fe700 1 -- 172.21.15.53:0/2351029457 wait complete. 2023-07-14T19:23:07.427 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 1 2023-07-14T19:23:07.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:07 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3056909355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:07.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:07 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3688352522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:23:07.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:07 smithi104 ceph-mon[98435]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:07.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:07 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2351029457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:07.762 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:23:07 smithi104 podman[123897]: 2023-07-14T19:23:07.762 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:23:07 smithi104 bash[123897]: c3518317168533cf172bd609cc83693c9eb35040bd9cdbd96538268e681702bc 2023-07-14T19:23:07.763 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:23:07 smithi104 ceph-mds[123937]: -- 172.21.15.104:0/742449998 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e140f5b1e0 con 0x55e140f30c00 2023-07-14T19:23:07.763 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:23:07 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-l[123913]: starting mds.l at 2023-07-14T19:23:07.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:07 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3056909355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:07.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:07 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3688352522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:23:07.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:07 smithi053 ceph-mon[96565]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:07.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:07 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2351029457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:07.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:07 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3056909355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:07.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:07 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3688352522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:23:07.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:07 smithi070 ceph-mon[98015]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:07.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:07 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2351029457' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:08.260 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:23:07 smithi104 systemd[1]: Started Ceph mds.l for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:08.774 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.104:6832/947644014,v1:172.21.15.104:6833/947644014] up:boot 2023-07-14T19:23:08.775 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: fsmap 1 up:standby 2023-07-14T19:23:08.775 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:23:08.775 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:08.775 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:08.775 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:08.775 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:08.775 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:08.776 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:08.776 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:08 smithi070 ceph-mon[98015]: Deploying daemon mds.k on smithi070 2023-07-14T19:23:09.010 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.104:6832/947644014,v1:172.21.15.104:6833/947644014] up:boot 2023-07-14T19:23:09.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: fsmap 1 up:standby 2023-07-14T19:23:09.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:23:09.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:09.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:09.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:09.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:09.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:09.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:09.012 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:08 smithi104 ceph-mon[98435]: Deploying daemon mds.k on smithi070 2023-07-14T19:23:09.071 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.104:6832/947644014,v1:172.21.15.104:6833/947644014] up:boot 2023-07-14T19:23:09.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: fsmap 1 up:standby 2023-07-14T19:23:09.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:23:09.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:09.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:09.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:09.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:09.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:09.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:09.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:08 smithi053 ceph-mon[96565]: Deploying daemon mds.k on smithi070 2023-07-14T19:23:09.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:09.478 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:23:09 smithi070 systemd[1]: Starting Ceph mds.k for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.577+0000 7f5d33a3c700 1 -- 172.21.15.53:0/1287763321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c100680 msgr2=0x7f5d2c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.577+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/1287763321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c100680 0x7f5d2c106f40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400b0a0 tx=0x7f5d2400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.577+0000 7f5d33a3c700 1 -- 172.21.15.53:0/1287763321 shutdown_connections 2023-07-14T19:23:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.577+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/1287763321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c100680 0x7f5d2c106f40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.577+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/1287763321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2c0feb00 0x7f5d2c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.577+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/1287763321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2c0ffcf0 0x7f5d2c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.577+0000 7f5d33a3c700 1 -- 172.21.15.53:0/1287763321 >> 172.21.15.53:0/1287763321 conn(0x7f5d2c0fa100 msgr2=0x7f5d2c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 -- 172.21.15.53:0/1287763321 shutdown_connections 2023-07-14T19:23:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 -- 172.21.15.53:0/1287763321 wait complete. 2023-07-14T19:23:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 Processor -- start 2023-07-14T19:23:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 -- start start 2023-07-14T19:23:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c0feb00 0x7f5d2c1947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2c0ffcf0 0x7f5d2c194d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:09.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2c100680 0x7f5d2c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:09.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c19c360 con 0x7f5d2c0feb00 2023-07-14T19:23:09.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c19c4a0 con 0x7f5d2c100680 2023-07-14T19:23:09.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d33a3c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c19c630 con 0x7f5d2c0ffcf0 2023-07-14T19:23:09.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d317d8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c0feb00 0x7f5d2c1947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:09.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.578+0000 7f5d317d8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c0feb00 0x7f5d2c1947e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49918/0 (socket says 172.21.15.53:49918) 2023-07-14T19:23:09.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d317d8700 1 -- 172.21.15.53:0/3787071213 learned_addr learned my addr 172.21.15.53:0/3787071213 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:09.583 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d30fd7700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2c0ffcf0 0x7f5d2c194d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:09.583 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d31fd9700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2c100680 0x7f5d2c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:09.584 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d317d8700 1 -- 172.21.15.53:0/3787071213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2c0ffcf0 msgr2=0x7f5d2c194d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:09.584 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d317d8700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2c0ffcf0 0x7f5d2c194d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.584 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d317d8700 1 -- 172.21.15.53:0/3787071213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2c100680 msgr2=0x7f5d2c19bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:09.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d317d8700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2c100680 0x7f5d2c19bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d317d8700 1 -- 172.21.15.53:0/3787071213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d24009d00 con 0x7f5d2c0feb00 2023-07-14T19:23:09.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d31fd9700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2c100680 0x7f5d2c19bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:23:09.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d317d8700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c0feb00 0x7f5d2c1947e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800c990 tx=0x7f5d2800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:09.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d227fc700 1 -- 172.21.15.53:0/3787071213 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28007b00 con 0x7f5d2c0feb00 2023-07-14T19:23:09.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d227fc700 1 -- 172.21.15.53:0/3787071213 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d28007c60 con 0x7f5d2c0feb00 2023-07-14T19:23:09.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d227fc700 1 -- 172.21.15.53:0/3787071213 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28018660 con 0x7f5d2c0feb00 2023-07-14T19:23:09.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c19c910 con 0x7f5d2c0feb00 2023-07-14T19:23:09.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.579+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c19cdd0 con 0x7f5d2c0feb00 2023-07-14T19:23:09.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.580+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c105ed0 con 0x7f5d2c0feb00 2023-07-14T19:23:09.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.581+0000 7f5d227fc700 1 -- 172.21.15.53:0/3787071213 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5d2800f450 con 0x7f5d2c0feb00 2023-07-14T19:23:09.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.582+0000 7f5d227fc700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d180a11a0 0x7f5d180a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:09.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.582+0000 7f5d30fd7700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d180a11a0 0x7f5d180a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:09.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.582+0000 7f5d227fc700 1 -- 172.21.15.53:0/3787071213 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5d280cc520 con 0x7f5d2c0feb00 2023-07-14T19:23:09.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.582+0000 7f5d30fd7700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d180a11a0 0x7f5d180a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c005ea0 tx=0x7f5d1c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:09.588 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.586+0000 7f5d227fc700 1 -- 172.21.15.53:0/3787071213 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5d28096120 con 0x7f5d2c0feb00 2023-07-14T19:23:09.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:09 smithi053 ceph-mon[96565]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:09.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.834+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d2c02c8e0 con 0x7f5d2c0feb00 2023-07-14T19:23:09.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.834+0000 7f5d227fc700 1 -- 172.21.15.53:0/3787071213 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1257 (secure 0 0 0) 0x7f5d28099740 con 0x7f5d2c0feb00 2023-07-14T19:23:09.837 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:09.837 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-14T19:23:09.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d180a11a0 msgr2=0x7f5d180a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:09.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d180a11a0 0x7f5d180a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c005ea0 tx=0x7f5d1c005e30 comp rx=0 tx=0).stop 2023-07-14T19:23:09.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c0feb00 msgr2=0x7f5d2c1947e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:09.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c0feb00 0x7f5d2c1947e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800c990 tx=0x7f5d2800cd50 comp rx=0 tx=0).stop 2023-07-14T19:23:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 shutdown_connections 2023-07-14T19:23:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d180a11a0 0x7f5d180a3650 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2c100680 0x7f5d2c19bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c0feb00 0x7f5d2c1947e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 --2- 172.21.15.53:0/3787071213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d2c0ffcf0 0x7f5d2c194d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.836+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 >> 172.21.15.53:0/3787071213 conn(0x7f5d2c0fa100 msgr2=0x7f5d2c103010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.837+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 shutdown_connections 2023-07-14T19:23:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:09.837+0000 7f5d33a3c700 1 -- 172.21.15.53:0/3787071213 wait complete. 2023-07-14T19:23:09.840 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 2 2023-07-14T19:23:09.964 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:09 smithi070 ceph-mon[98015]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:09.965 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:23:09 smithi070 podman[122369]: 2023-07-14T19:23:09.965 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:23:09 smithi070 ceph-mds[122409]: -- 172.21.15.70:0/914896388 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d8347ef1e0 con 0x55d8347c4800 2023-07-14T19:23:09.965 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:23:09 smithi070 bash[122369]: 72d94f7f2e55ada4d9d85d7cecaa769a9713212af4ccaa9cf0b25d3f2de69107 2023-07-14T19:23:09.966 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:23:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-k[122384]: starting mds.k at 2023-07-14T19:23:10.010 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:09 smithi104 ceph-mon[98435]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:10.227 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:23:09 smithi070 systemd[1]: Started Ceph mds.k for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:10.973 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:10 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3787071213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:10.973 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:10.973 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:10.974 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:10.974 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:10.974 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:10.974 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:10 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:10.974 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:10 smithi053 ceph-mon[96565]: Deploying daemon mds.j on smithi053 2023-07-14T19:23:10.977 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:10 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3787071213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:10.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:10.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:10.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:10.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:10.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:10.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:10 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:10.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:10 smithi070 ceph-mon[98015]: Deploying daemon mds.j on smithi053 2023-07-14T19:23:11.010 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:10 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3787071213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:11.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:11.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:11.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:11.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:11.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:11.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:10 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:11.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:10 smithi104 ceph-mon[98435]: Deploying daemon mds.j on smithi053 2023-07-14T19:23:11.572 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:23:11 smithi053 systemd[1]: Starting Ceph mds.j for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:11.851 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:11.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:11 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6832/3920144890,v1:172.21.15.70:6833/3920144890] up:boot 2023-07-14T19:23:11.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:11 smithi070 ceph-mon[98015]: fsmap 2 up:standby 2023-07-14T19:23:11.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:11 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:23:11.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:11 smithi070 ceph-mon[98015]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 -- 172.21.15.53:0/1288240257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34000feb00 msgr2=0x7f34000fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 --2- 172.21.15.53:0/1288240257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34000feb00 0x7f34000fef10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009990 tx=0x7f33f4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 -- 172.21.15.53:0/1288240257 shutdown_connections 2023-07-14T19:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 --2- 172.21.15.53:0/1288240257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3400100680 0x7f3400106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 --2- 172.21.15.53:0/1288240257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34000feb00 0x7f34000fef10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 --2- 172.21.15.53:0/1288240257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34000ffcf0 0x7f3400100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 -- 172.21.15.53:0/1288240257 >> 172.21.15.53:0/1288240257 conn(0x7f34000fa100 msgr2=0x7f34000fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 -- 172.21.15.53:0/1288240257 shutdown_connections 2023-07-14T19:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 -- 172.21.15.53:0/1288240257 wait complete. 2023-07-14T19:23:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 Processor -- start 2023-07-14T19:23:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.996+0000 7f3406b09700 1 -- start start 2023-07-14T19:23:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f3406b09700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34000feb00 0x7f3400195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f3406b09700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34000ffcf0 0x7f3400195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f3406b09700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3400100680 0x7f340019cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f3406b09700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340019d4d0 con 0x7f3400100680 2023-07-14T19:23:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f3406b09700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340019d610 con 0x7f34000feb00 2023-07-14T19:23:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f3406b09700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340019d7a0 con 0x7f34000ffcf0 2023-07-14T19:23:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34050a6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3400100680 0x7f340019cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:12.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34050a6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3400100680 0x7f340019cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49936/0 (socket says 172.21.15.53:49936) 2023-07-14T19:23:12.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34050a6700 1 -- 172.21.15.53:0/1453998473 learned_addr learned my addr 172.21.15.53:0/1453998473 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:12.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34048a5700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34000feb00 0x7f3400195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:12.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f33fffff700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34000ffcf0 0x7f3400195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:12.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34050a6700 1 -- 172.21.15.53:0/1453998473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34000ffcf0 msgr2=0x7f3400195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34050a6700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34000ffcf0 0x7f3400195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34050a6700 1 -- 172.21.15.53:0/1453998473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34000feb00 msgr2=0x7f3400195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34050a6700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34000feb00 0x7f3400195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34050a6700 1 -- 172.21.15.53:0/1453998473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f40095f0 con 0x7f3400100680 2023-07-14T19:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f33fffff700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34000ffcf0 0x7f3400195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f34050a6700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3400100680 0x7f340019cf00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f33f8007ee0 tx=0x7f33f800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f33fdffb700 1 -- 172.21.15.53:0/1453998473 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f800ddd0 con 0x7f3400100680 2023-07-14T19:23:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f33fdffb700 1 -- 172.21.15.53:0/1453998473 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33f800f040 con 0x7f3400100680 2023-07-14T19:23:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340019da80 con 0x7f3400100680 2023-07-14T19:23:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.997+0000 7f33fdffb700 1 -- 172.21.15.53:0/1453998473 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f80048e0 con 0x7f3400100680 2023-07-14T19:23:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.998+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f340019df40 con 0x7f3400100680 2023-07-14T19:23:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.999+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340010a320 con 0x7f3400100680 2023-07-14T19:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:11.999+0000 7f33fdffb700 1 -- 172.21.15.53:0/1453998473 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f33f8003a00 con 0x7f3400100680 2023-07-14T19:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.000+0000 7f33fdffb700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33f00a11a0 0x7f33f00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.000+0000 7f34048a5700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33f00a11a0 0x7f33f00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.000+0000 7f33fdffb700 1 -- 172.21.15.53:0/1453998473 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f33f80cbe40 con 0x7f3400100680 2023-07-14T19:23:12.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.001+0000 7f34048a5700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33f00a11a0 0x7f33f00a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f33f400b5c0 tx=0x7f33f4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:12.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.005+0000 7f33fdffb700 1 -- 172.21.15.53:0/1453998473 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f33f804e780 con 0x7f3400100680 2023-07-14T19:23:12.010 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:11 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6832/3920144890,v1:172.21.15.70:6833/3920144890] up:boot 2023-07-14T19:23:12.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:11 smithi104 ceph-mon[98435]: fsmap 2 up:standby 2023-07-14T19:23:12.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:11 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:23:12.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:11 smithi104 ceph-mon[98435]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:12.071 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:23:11 smithi053 podman[129975]: 2023-07-14T19:23:12.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:11 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6832/3920144890,v1:172.21.15.70:6833/3920144890] up:boot 2023-07-14T19:23:12.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:11 smithi053 ceph-mon[96565]: fsmap 2 up:standby 2023-07-14T19:23:12.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:11 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:23:12.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:11 smithi053 ceph-mon[96565]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:12.268 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.266+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f340019e550 con 0x7f3400100680 2023-07-14T19:23:12.269 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.267+0000 7f33fdffb700 1 -- 172.21.15.53:0/1453998473 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2014 (secure 0 0 0) 0x7f33f8095990 con 0x7f3400100680 2023-07-14T19:23:12.269 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:12.269 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:23:12.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33f00a11a0 msgr2=0x7f33f00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:12.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33f00a11a0 0x7f33f00a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f33f400b5c0 tx=0x7f33f4005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:12.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3400100680 msgr2=0x7f340019cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:12.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3400100680 0x7f340019cf00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f33f8007ee0 tx=0x7f33f800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:12.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 shutdown_connections 2023-07-14T19:23:12.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33f00a11a0 0x7f33f00a3650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:12.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3400100680 0x7f340019cf00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:12.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34000feb00 0x7f3400195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:12.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 --2- 172.21.15.53:0/1453998473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34000ffcf0 0x7f3400195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:12.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 >> 172.21.15.53:0/1453998473 conn(0x7f34000fa100 msgr2=0x7f3400067860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:12.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 shutdown_connections 2023-07-14T19:23:12.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:12.270+0000 7f3406b09700 1 -- 172.21.15.53:0/1453998473 wait complete. 2023-07-14T19:23:12.273 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 3 2023-07-14T19:23:12.573 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:23:12 smithi053 bash[129975]: 50f1864c386355edf4604751e6f692a1326595391d6a5cbb13b9058322a9cd18 2023-07-14T19:23:12.573 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:23:12 smithi053 ceph-mds[130059]: -- 172.21.15.53:0/2072181513 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5596623391e0 con 0x55966230ec00 2023-07-14T19:23:12.573 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:23:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-j[130001]: starting mds.j at 2023-07-14T19:23:12.574 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:23:12 smithi053 systemd[1]: Started Ceph mds.j for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:13.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:13.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:13.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:13.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:13.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:13.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:13.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: Deploying daemon mds.d on smithi053 2023-07-14T19:23:13.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1453998473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:13.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.53:6834/2670019040,v1:172.21.15.53:6835/2670019040] up:boot 2023-07-14T19:23:13.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: fsmap 3 up:standby 2023-07-14T19:23:13.413 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:13 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:23:13.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:13.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:13.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:13.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:13.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:13.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:13.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: Deploying daemon mds.d on smithi053 2023-07-14T19:23:13.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1453998473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:13.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.53:6834/2670019040,v1:172.21.15.53:6835/2670019040] up:boot 2023-07-14T19:23:13.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: fsmap 3 up:standby 2023-07-14T19:23:13.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:13 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:23:13.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:13.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:13.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:13.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:13.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:13.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:13.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: Deploying daemon mds.d on smithi053 2023-07-14T19:23:13.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1453998473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:13.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.53:6834/2670019040,v1:172.21.15.53:6835/2670019040] up:boot 2023-07-14T19:23:13.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: fsmap 3 up:standby 2023-07-14T19:23:13.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:13 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:23:13.663 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:23:13 smithi053 systemd[1]: Starting Ceph mds.d for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:14.215 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:23:13 smithi053 podman[130364]: 2023-07-14T19:23:14.216 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:23:14 smithi053 bash[130364]: 095b13036b09f322bb815e6b253a6f2713fcdd198b83b36436892b7336915b95 2023-07-14T19:23:14.216 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:23:14 smithi053 ceph-mds[130404]: -- 172.21.15.53:0/1078029608 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cfc44c51e0 con 0x55cfc449a800 2023-07-14T19:23:14.216 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:14 smithi053 ceph-mon[96565]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:14.286 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:14.422 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.418+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/1247672834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 msgr2=0x7f0f440fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:14.423 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.418+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/1247672834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 0x7f0f440fef10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009990 tx=0x7f0f40009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.423 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.418+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/1247672834 shutdown_connections 2023-07-14T19:23:14.423 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.418+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/1247672834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f44100680 0x7f0f44106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.418+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/1247672834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 0x7f0f440fef10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.418+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/1247672834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f440ffcf0 0x7f0f44100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.418+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/1247672834 >> 172.21.15.53:0/1247672834 conn(0x7f0f440fa100 msgr2=0x7f0f440fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:14.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.418+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/1247672834 shutdown_connections 2023-07-14T19:23:14.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.418+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/1247672834 wait complete. 2023-07-14T19:23:14.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4c2f9700 1 Processor -- start 2023-07-14T19:23:14.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4c2f9700 1 -- start start 2023-07-14T19:23:14.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4c2f9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 0x7f0f44195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:14.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4c2f9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f440ffcf0 0x7f0f44195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:14.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4c2f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f44100680 0x7f0f4419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:14.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4c2f9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4419d4d0 con 0x7f0f440feb00 2023-07-14T19:23:14.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4c2f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4419d610 con 0x7f0f44100680 2023-07-14T19:23:14.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4c2f9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4419d7a0 con 0x7f0f440ffcf0 2023-07-14T19:23:14.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4a095700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 0x7f0f44195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:14.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4a095700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 0x7f0f44195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49970/0 (socket says 172.21.15.53:49970) 2023-07-14T19:23:14.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4a095700 1 -- 172.21.15.53:0/256150738 learned_addr learned my addr 172.21.15.53:0/256150738 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:14.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.419+0000 7f0f4a896700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f44100680 0x7f0f4419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:14.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f49894700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f440ffcf0 0x7f0f44195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:14.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f4a095700 1 -- 172.21.15.53:0/256150738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f440ffcf0 msgr2=0x7f0f44195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:14.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f4a095700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f440ffcf0 0x7f0f44195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f4a095700 1 -- 172.21.15.53:0/256150738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f44100680 msgr2=0x7f0f4419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:14.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f4a095700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f44100680 0x7f0f4419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f4a095700 1 -- 172.21.15.53:0/256150738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f400095f0 con 0x7f0f440feb00 2023-07-14T19:23:14.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f4a095700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 0x7f0f44195950 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0f400098e0 tx=0x7f0f40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:14.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f3b7fe700 1 -- 172.21.15.53:0/256150738 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4001d070 con 0x7f0f440feb00 2023-07-14T19:23:14.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4419da20 con 0x7f0f440feb00 2023-07-14T19:23:14.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f3b7fe700 1 -- 172.21.15.53:0/256150738 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f400054f0 con 0x7f0f440feb00 2023-07-14T19:23:14.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f3b7fe700 1 -- 172.21.15.53:0/256150738 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f40004ca0 con 0x7f0f440feb00 2023-07-14T19:23:14.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.420+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4419dde0 con 0x7f0f440feb00 2023-07-14T19:23:14.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.421+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f44106b90 con 0x7f0f440feb00 2023-07-14T19:23:14.430 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.422+0000 7f0f3b7fe700 1 -- 172.21.15.53:0/256150738 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0f4000fb90 con 0x7f0f440feb00 2023-07-14T19:23:14.431 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.423+0000 7f0f3b7fe700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0f300a11a0 0x7f0f300a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:14.431 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.423+0000 7f0f49894700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0f300a11a0 0x7f0f300a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:14.431 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.423+0000 7f0f3b7fe700 1 -- 172.21.15.53:0/256150738 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0f400cdb70 con 0x7f0f440feb00 2023-07-14T19:23:14.431 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.424+0000 7f0f49894700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0f300a11a0 0x7f0f300a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0f34005fd0 tx=0x7f0f34005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:14.431 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.427+0000 7f0f3b7fe700 1 -- 172.21.15.53:0/256150738 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f400977f0 con 0x7f0f440feb00 2023-07-14T19:23:14.477 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:14 smithi070 ceph-mon[98015]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:14.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:14 smithi104 ceph-mon[98435]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:14.571 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:23:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-d[130380]: starting mds.d at 2023-07-14T19:23:14.572 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:23:14 smithi053 systemd[1]: Started Ceph mds.d for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:14.672 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.671+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f4402cb50 con 0x7f0f440feb00 2023-07-14T19:23:14.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.671+0000 7f0f3b7fe700 1 -- 172.21.15.53:0/256150738 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2771 (secure 0 0 0) 0x7f0f4009ae10 con 0x7f0f440feb00 2023-07-14T19:23:14.674 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:14.674 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:23:14.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0f300a11a0 msgr2=0x7f0f300a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:14.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0f300a11a0 0x7f0f300a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0f34005fd0 tx=0x7f0f34005f60 comp rx=0 tx=0).stop 2023-07-14T19:23:14.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 msgr2=0x7f0f44195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:14.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 0x7f0f44195950 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0f400098e0 tx=0x7f0f40004730 comp rx=0 tx=0).stop 2023-07-14T19:23:14.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 shutdown_connections 2023-07-14T19:23:14.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0f300a11a0 0x7f0f300a3650 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f44100680 0x7f0f4419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f440feb00 0x7f0f44195950 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 --2- 172.21.15.53:0/256150738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f440ffcf0 0x7f0f44195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 >> 172.21.15.53:0/256150738 conn(0x7f0f440fa100 msgr2=0x7f0f440fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:14.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 shutdown_connections 2023-07-14T19:23:14.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:14.673+0000 7f0f4c2f9700 1 -- 172.21.15.53:0/256150738 wait complete. 2023-07-14T19:23:14.678 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 4 2023-07-14T19:23:15.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:15.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/256150738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:15.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.53:6836/1144651849,v1:172.21.15.53:6837/1144651849] up:boot 2023-07-14T19:23:15.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: fsmap 4 up:standby 2023-07-14T19:23:15.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:15 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:23:15.608 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:15.609 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:15.609 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:15.609 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:15.609 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:15.609 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:15.609 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/256150738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:15.610 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.53:6836/1144651849,v1:172.21.15.53:6837/1144651849] up:boot 2023-07-14T19:23:15.610 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: fsmap 4 up:standby 2023-07-14T19:23:15.610 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:15 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:23:15.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:15.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:15.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:15.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:15.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:15.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:15.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/256150738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:15.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.53:6836/1144651849,v1:172.21.15.53:6837/1144651849] up:boot 2023-07-14T19:23:15.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: fsmap 4 up:standby 2023-07-14T19:23:15.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:15 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:23:15.978 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:23:15 smithi070 systemd[1]: Starting Ceph mds.e for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:16.436 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:16 smithi070 ceph-mon[98015]: Deploying daemon mds.e on smithi070 2023-07-14T19:23:16.436 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:16 smithi070 ceph-mon[98015]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:16.437 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:23:16 smithi070 podman[122708]: 2023-07-14T19:23:16.437 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:23:16 smithi070 ceph-mds[122748]: -- 172.21.15.70:0/1564114705 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564d665091e0 con 0x564d664df000 2023-07-14T19:23:16.437 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:23:16 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-e[122724]: starting mds.e at 2023-07-14T19:23:16.437 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:23:16 smithi070 bash[122708]: 2d1155c1a53f09025c82714b178e3f45654ddc38d487dafbe707b9db865e503d 2023-07-14T19:23:16.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:16 smithi053 ceph-mon[96565]: Deploying daemon mds.e on smithi070 2023-07-14T19:23:16.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:16 smithi053 ceph-mon[96565]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:16.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:16.728 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:23:16 smithi070 systemd[1]: Started Ceph mds.e for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:16.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:16 smithi104 ceph-mon[98435]: Deploying daemon mds.e on smithi070 2023-07-14T19:23:16.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:16 smithi104 ceph-mon[98435]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 -- 172.21.15.53:0/3360450694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb800feb00 msgr2=0x7ffb800fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:16.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 --2- 172.21.15.53:0/3360450694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb800feb00 0x7ffb800fef10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009990 tx=0x7ffb70009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:16.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 -- 172.21.15.53:0/3360450694 shutdown_connections 2023-07-14T19:23:16.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 --2- 172.21.15.53:0/3360450694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb80100680 0x7ffb80106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 --2- 172.21.15.53:0/3360450694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb800feb00 0x7ffb800fef10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 --2- 172.21.15.53:0/3360450694 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb800ffcf0 0x7ffb80100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 -- 172.21.15.53:0/3360450694 >> 172.21.15.53:0/3360450694 conn(0x7ffb800fa100 msgr2=0x7ffb800fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 -- 172.21.15.53:0/3360450694 shutdown_connections 2023-07-14T19:23:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 -- 172.21.15.53:0/3360450694 wait complete. 2023-07-14T19:23:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 Processor -- start 2023-07-14T19:23:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 -- start start 2023-07-14T19:23:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.828+0000 7ffb85d60700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb800feb00 0x7ffb80068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb85d60700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb800ffcf0 0x7ffb8019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb85d60700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb80100680 0x7ffb8019cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb85d60700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019d560 con 0x7ffb80100680 2023-07-14T19:23:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb85d60700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019d6d0 con 0x7ffb800ffcf0 2023-07-14T19:23:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb85d60700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019d890 con 0x7ffb800feb00 2023-07-14T19:23:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb80100680 0x7ffb8019cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb80100680 0x7ffb8019cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49990/0 (socket says 172.21.15.53:49990) 2023-07-14T19:23:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb77fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb800ffcf0 0x7ffb8019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:16.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7ffff700 1 -- 172.21.15.53:0/223091575 learned_addr learned my addr 172.21.15.53:0/223091575 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:16.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb800feb00 0x7ffb80068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:16.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7ffff700 1 -- 172.21.15.53:0/223091575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb800feb00 msgr2=0x7ffb80068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:16.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7ffff700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb800feb00 0x7ffb80068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:16.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7ffff700 1 -- 172.21.15.53:0/223091575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb800ffcf0 msgr2=0x7ffb8019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:16.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7ffff700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb800ffcf0 0x7ffb8019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:16.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7ffff700 1 -- 172.21.15.53:0/223091575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb78009c40 con 0x7ffb80100680 2023-07-14T19:23:16.843 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb77fff700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb800ffcf0 0x7ffb8019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:23:16.843 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7ffff700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb80100680 0x7ffb8019cd90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800c370 tx=0x7ffb7800c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:16.843 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7d7fa700 1 -- 172.21.15.53:0/223091575 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78017070 con 0x7ffb80100680 2023-07-14T19:23:16.844 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7d7fa700 1 -- 172.21.15.53:0/223091575 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb7800f040 con 0x7ffb80100680 2023-07-14T19:23:16.844 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb700095f0 con 0x7ffb80100680 2023-07-14T19:23:16.845 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb7d7fa700 1 -- 172.21.15.53:0/223091575 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7800e050 con 0x7ffb80100680 2023-07-14T19:23:16.845 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.829+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8019de90 con 0x7ffb80100680 2023-07-14T19:23:16.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.830+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb801060e0 con 0x7ffb80100680 2023-07-14T19:23:16.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.832+0000 7ffb7d7fa700 1 -- 172.21.15.53:0/223091575 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ffb7800cb60 con 0x7ffb80100680 2023-07-14T19:23:16.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.833+0000 7ffb7d7fa700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:16.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.833+0000 7ffb7d7fa700 1 -- 172.21.15.53:0/223091575 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffb780cc3b0 con 0x7ffb80100680 2023-07-14T19:23:16.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.833+0000 7ffb7f7fe700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:16.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.833+0000 7ffb7f7fe700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000b2e0 tx=0x7ffb70005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:16.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:16.836+0000 7ffb7d7fa700 1 -- 172.21.15.53:0/223091575 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ffb78095fa0 con 0x7ffb80100680 2023-07-14T19:23:17.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.080+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb8002c8e0 con 0x7ffb80100680 2023-07-14T19:23:17.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.080+0000 7ffb7d7fa700 1 -- 172.21.15.53:0/223091575 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3528 (secure 0 0 0) 0x7ffb780995c0 con 0x7ffb80100680 2023-07-14T19:23:17.082 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:17.082 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:23:17.084 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ffb6c0a11a0 msgr2=0x7ffb6c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:17.084 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000b2e0 tx=0x7ffb70005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:17.084 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb80100680 msgr2=0x7ffb8019cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:17.084 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb80100680 0x7ffb8019cd90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffb7800c370 tx=0x7ffb7800c680 comp rx=0 tx=0).stop 2023-07-14T19:23:17.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 shutdown_connections 2023-07-14T19:23:17.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ffb6c0a11a0 0x7ffb6c0a3650 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:17.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb80100680 0x7ffb8019cd90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:17.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb800feb00 0x7ffb80068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:17.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 --2- 172.21.15.53:0/223091575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb800ffcf0 0x7ffb8019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:17.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 >> 172.21.15.53:0/223091575 conn(0x7ffb800fa100 msgr2=0x7ffb80103220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:17.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 shutdown_connections 2023-07-14T19:23:17.086 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:17.082+0000 7ffb85d60700 1 -- 172.21.15.53:0/223091575 wait complete. 2023-07-14T19:23:17.086 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 5 2023-07-14T19:23:17.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:17.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:17.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:17.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:17.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:17.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:17.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: Deploying daemon mds.f on smithi104 2023-07-14T19:23:17.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/223091575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:17.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] up:boot 2023-07-14T19:23:17.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: fsmap 5 up:standby 2023-07-14T19:23:17.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:23:17.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:17 smithi104 ceph-mon[98435]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:17.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:17.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:17.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:17.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:17.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:17.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:17.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: Deploying daemon mds.f on smithi104 2023-07-14T19:23:17.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/223091575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:17.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] up:boot 2023-07-14T19:23:17.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: fsmap 5 up:standby 2023-07-14T19:23:17.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:23:17.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:17 smithi053 ceph-mon[96565]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:17.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:17.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:17.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:17.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:17.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:17.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:17.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: Deploying daemon mds.f on smithi104 2023-07-14T19:23:17.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/223091575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:17.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] up:boot 2023-07-14T19:23:17.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: fsmap 5 up:standby 2023-07-14T19:23:17.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:23:17.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:17 smithi070 ceph-mon[98015]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:18.261 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:23:18 smithi104 systemd[1]: Starting Ceph mds.f for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:18.588 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:23:18 smithi104 podman[124236]: 2023-07-14T19:23:18.588 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:23:18 smithi104 ceph-mds[124276]: -- 172.21.15.104:0/2894888950 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5594631b71e0 con 0x55946318d000 2023-07-14T19:23:18.589 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:23:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-f[124252]: starting mds.f at 2023-07-14T19:23:18.589 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:23:18 smithi104 bash[124236]: ddc30f5dbf7341c1b63e989da1cf40ce487d560296470bdb6d1b24e2d77cc8af 2023-07-14T19:23:19.010 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:23:18 smithi104 systemd[1]: Started Ceph mds.f for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:19.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:19.238 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.235+0000 7faa64967700 1 -- 172.21.15.53:0/3217787665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600feb00 msgr2=0x7faa600fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:19.238 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.235+0000 7faa64967700 1 --2- 172.21.15.53:0/3217787665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600feb00 0x7faa600fef10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faa50009990 tx=0x7faa50009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.238 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.236+0000 7faa64967700 1 -- 172.21.15.53:0/3217787665 shutdown_connections 2023-07-14T19:23:19.238 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.236+0000 7faa64967700 1 --2- 172.21.15.53:0/3217787665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa60100680 0x7faa60106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.238 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.236+0000 7faa64967700 1 --2- 172.21.15.53:0/3217787665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600feb00 0x7faa600fef10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.236+0000 7faa64967700 1 --2- 172.21.15.53:0/3217787665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa600ffcf0 0x7faa60100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.236+0000 7faa64967700 1 -- 172.21.15.53:0/3217787665 >> 172.21.15.53:0/3217787665 conn(0x7faa600fa100 msgr2=0x7faa600fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:19.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.236+0000 7faa64967700 1 -- 172.21.15.53:0/3217787665 shutdown_connections 2023-07-14T19:23:19.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.236+0000 7faa64967700 1 -- 172.21.15.53:0/3217787665 wait complete. 2023-07-14T19:23:19.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.236+0000 7faa64967700 1 Processor -- start 2023-07-14T19:23:19.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa64967700 1 -- start start 2023-07-14T19:23:19.241 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa64967700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa600feb00 0x7faa60194810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:19.241 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa64967700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600ffcf0 0x7faa60194d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:19.241 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa64967700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa60100680 0x7faa6019bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:19.241 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa64967700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6019c390 con 0x7faa600ffcf0 2023-07-14T19:23:19.242 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa64967700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6019c4d0 con 0x7faa600feb00 2023-07-14T19:23:19.242 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa64967700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6019c660 con 0x7faa60100680 2023-07-14T19:23:19.242 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600ffcf0 0x7faa60194d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:19.242 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600ffcf0 0x7faa60194d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50002/0 (socket says 172.21.15.53:50002) 2023-07-14T19:23:19.242 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5dd9b700 1 -- 172.21.15.53:0/2744148000 learned_addr learned my addr 172.21.15.53:0/2744148000 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:19.242 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5ed9d700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa60100680 0x7faa6019bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:19.242 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5e59c700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa600feb00 0x7faa60194810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:19.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5dd9b700 1 -- 172.21.15.53:0/2744148000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa60100680 msgr2=0x7faa6019bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:19.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5dd9b700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa60100680 0x7faa6019bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5dd9b700 1 -- 172.21.15.53:0/2744148000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa600feb00 msgr2=0x7faa60194810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:19.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5dd9b700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa600feb00 0x7faa60194810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5dd9b700 1 -- 172.21.15.53:0/2744148000 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa500095f0 con 0x7faa600ffcf0 2023-07-14T19:23:19.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.237+0000 7faa5dd9b700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600ffcf0 0x7faa60194d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faa5400e9a0 tx=0x7faa5400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:19.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.238+0000 7faa4f7fe700 1 -- 172.21.15.53:0/2744148000 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54009790 con 0x7faa600ffcf0 2023-07-14T19:23:19.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.238+0000 7faa4f7fe700 1 -- 172.21.15.53:0/2744148000 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa5400cbd0 con 0x7faa600ffcf0 2023-07-14T19:23:19.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.238+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6019c940 con 0x7faa600ffcf0 2023-07-14T19:23:19.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.238+0000 7faa4f7fe700 1 -- 172.21.15.53:0/2744148000 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54018680 con 0x7faa600ffcf0 2023-07-14T19:23:19.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.238+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6019ce50 con 0x7faa600ffcf0 2023-07-14T19:23:19.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.239+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa601060a0 con 0x7faa600ffcf0 2023-07-14T19:23:19.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.240+0000 7faa4f7fe700 1 -- 172.21.15.53:0/2744148000 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faa5400cd40 con 0x7faa600ffcf0 2023-07-14T19:23:19.245 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.241+0000 7faa4f7fe700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faa480a11a0 0x7faa480a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:19.245 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.241+0000 7faa5e59c700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faa480a11a0 0x7faa480a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:19.245 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.241+0000 7faa4f7fe700 1 -- 172.21.15.53:0/2744148000 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faa54014070 con 0x7faa600ffcf0 2023-07-14T19:23:19.245 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.241+0000 7faa5e59c700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faa480a11a0 0x7faa480a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faa50006010 tx=0x7faa5000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:19.246 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.244+0000 7faa4f7fe700 1 -- 172.21.15.53:0/2744148000 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faa54096590 con 0x7faa600ffcf0 2023-07-14T19:23:19.492 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.491+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa6002caf0 con 0x7faa600ffcf0 2023-07-14T19:23:19.493 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.491+0000 7faa4f7fe700 1 -- 172.21.15.53:0/2744148000 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5045 (secure 0 0 0) 0x7faa54094e60 con 0x7faa600ffcf0 2023-07-14T19:23:19.494 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:19.494 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T19:23:19.495 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faa480a11a0 msgr2=0x7faa480a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:19.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faa480a11a0 0x7faa480a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7faa50006010 tx=0x7faa5000b260 comp rx=0 tx=0).stop 2023-07-14T19:23:19.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600ffcf0 msgr2=0x7faa60194d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600ffcf0 0x7faa60194d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faa5400e9a0 tx=0x7faa5400ecb0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 shutdown_connections 2023-07-14T19:23:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faa480a11a0 0x7faa480a3650 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faa60100680 0x7faa6019bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa600feb00 0x7faa60194810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 --2- 172.21.15.53:0/2744148000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa600ffcf0 0x7faa60194d50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 >> 172.21.15.53:0/2744148000 conn(0x7faa600fa100 msgr2=0x7faa601031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.493+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 shutdown_connections 2023-07-14T19:23:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:19.494+0000 7faa64967700 1 -- 172.21.15.53:0/2744148000 wait complete. 2023-07-14T19:23:19.498 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2023-07-14T19:23:19.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:19.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:19.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:19.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:19.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:19.823 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:19.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: Deploying daemon mds.c on smithi104 2023-07-14T19:23:19.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.104:6834/1013468175,v1:172.21.15.104:6835/1013468175] up:boot 2023-07-14T19:23:19.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: fsmap 6 up:standby 2023-07-14T19:23:19.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:23:19.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:19.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:19 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2744148000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:19.927 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:19.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:19.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:19.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:19.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:19.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:19.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: Deploying daemon mds.c on smithi104 2023-07-14T19:23:19.928 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.104:6834/1013468175,v1:172.21.15.104:6835/1013468175] up:boot 2023-07-14T19:23:19.929 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: fsmap 6 up:standby 2023-07-14T19:23:19.929 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:23:19.929 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:19.929 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:19 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2744148000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:19.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:19.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:19.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:19.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:19.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:19.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:19.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: Deploying daemon mds.c on smithi104 2023-07-14T19:23:19.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.104:6834/1013468175,v1:172.21.15.104:6835/1013468175] up:boot 2023-07-14T19:23:19.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: fsmap 6 up:standby 2023-07-14T19:23:19.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:23:19.979 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:19.980 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:19 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2744148000' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:20.261 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:23:20 smithi104 systemd[1]: Starting Ceph mds.c for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:20.717 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:23:20 smithi104 podman[124580]: 2023-07-14T19:23:20.717 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:23:20 smithi104 bash[124580]: 40a7f7a2daa3c6bb39724a7992c4c54a7c270a48c740bf3ba36b35da0ab8d561 2023-07-14T19:23:21.010 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:23:20 smithi104 ceph-mds[124620]: -- 172.21.15.104:0/1939329176 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f20cda91e0 con 0x55f20cd7ec00 2023-07-14T19:23:21.011 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:23:20 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-c[124596]: starting mds.c at 2023-07-14T19:23:21.011 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:23:20 smithi104 systemd[1]: Started Ceph mds.c for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:21.511 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:21.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.650+0000 7f02d1de8700 1 -- 172.21.15.53:0/812075943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc0ffd20 msgr2=0x7f02cc100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:21.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.650+0000 7f02d1de8700 1 --2- 172.21.15.53:0/812075943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc0ffd20 0x7f02cc100170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f02b4009990 tx=0x7f02b4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.650+0000 7f02d1de8700 1 -- 172.21.15.53:0/812075943 shutdown_connections 2023-07-14T19:23:21.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.650+0000 7f02d1de8700 1 --2- 172.21.15.53:0/812075943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02cc1006b0 0x7f02cc106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.650+0000 7f02d1de8700 1 --2- 172.21.15.53:0/812075943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02cc0feb30 0x7f02cc0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.650+0000 7f02d1de8700 1 --2- 172.21.15.53:0/812075943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc0ffd20 0x7f02cc100170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.650+0000 7f02d1de8700 1 -- 172.21.15.53:0/812075943 >> 172.21.15.53:0/812075943 conn(0x7f02cc0fa150 msgr2=0x7f02cc0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:21.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.650+0000 7f02d1de8700 1 -- 172.21.15.53:0/812075943 shutdown_connections 2023-07-14T19:23:21.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.650+0000 7f02d1de8700 1 -- 172.21.15.53:0/812075943 wait complete. 2023-07-14T19:23:21.654 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02d1de8700 1 Processor -- start 2023-07-14T19:23:21.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02d1de8700 1 -- start start 2023-07-14T19:23:21.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02d1de8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02cc0feb30 0x7f02cc069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:21.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02d1de8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02cc0ffd20 0x7f02cc19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:21.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02d1de8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc1006b0 0x7f02cc19cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:21.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02d1de8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc19d420 con 0x7f02cc1006b0 2023-07-14T19:23:21.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02d1de8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc19d590 con 0x7f02cc0feb30 2023-07-14T19:23:21.656 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02d1de8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc19d750 con 0x7f02cc0ffd20 2023-07-14T19:23:21.656 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02cbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc1006b0 0x7f02cc19cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:21.656 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02cb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02cc0feb30 0x7f02cc069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:21.656 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02cbfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc1006b0 0x7f02cc19cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50016/0 (socket says 172.21.15.53:50016) 2023-07-14T19:23:21.656 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.651+0000 7f02cbfff700 1 -- 172.21.15.53:0/4249603701 learned_addr learned my addr 172.21.15.53:0/4249603701 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:21.656 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02caffd700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02cc0ffd20 0x7f02cc19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:21.656 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02cb7fe700 1 -- 172.21.15.53:0/4249603701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02cc0ffd20 msgr2=0x7f02cc19a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:21.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02cb7fe700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02cc0ffd20 0x7f02cc19a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02cb7fe700 1 -- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc1006b0 msgr2=0x7f02cc19cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:21.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02cb7fe700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc1006b0 0x7f02cc19cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02cb7fe700 1 -- 172.21.15.53:0/4249603701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c4009c40 con 0x7f02cc0feb30 2023-07-14T19:23:21.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02cbfff700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc1006b0 0x7f02cc19cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:23:21.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02cb7fe700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02cc0feb30 0x7f02cc069530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00d740 tx=0x7f02bc00da50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:21.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02c8ff9700 1 -- 172.21.15.53:0/4249603701 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc004220 con 0x7f02cc0feb30 2023-07-14T19:23:21.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b40095f0 con 0x7f02cc0feb30 2023-07-14T19:23:21.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02c8ff9700 1 -- 172.21.15.53:0/4249603701 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02bc004380 con 0x7f02cc0feb30 2023-07-14T19:23:21.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02c8ff9700 1 -- 172.21.15.53:0/4249603701 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc003cf0 con 0x7f02cc0feb30 2023-07-14T19:23:21.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.652+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02cc19ddb0 con 0x7f02cc0feb30 2023-07-14T19:23:21.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.653+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02cc1060a0 con 0x7f02cc0feb30 2023-07-14T19:23:21.659 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.654+0000 7f02c8ff9700 1 -- 172.21.15.53:0/4249603701 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f02bc00f690 con 0x7f02cc0feb30 2023-07-14T19:23:21.659 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.655+0000 7f02c8ff9700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f02b80a11a0 0x7f02b80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:21.659 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.655+0000 7f02c8ff9700 1 -- 172.21.15.53:0/4249603701 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f02bc0cc0a0 con 0x7f02cc0feb30 2023-07-14T19:23:21.659 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.655+0000 7f02caffd700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f02b80a11a0 0x7f02b80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:21.659 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.655+0000 7f02caffd700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f02b80a11a0 0x7f02b80a3650 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f02b4006010 tx=0x7f02b400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:21.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.662+0000 7f02c8ff9700 1 -- 172.21.15.53:0/4249603701 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f02bc095bf0 con 0x7f02cc0feb30 2023-07-14T19:23:21.912 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.910+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02cc02c8e0 con 0x7f02cc0feb30 2023-07-14T19:23:21.912 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.911+0000 7f02c8ff9700 1 -- 172.21.15.53:0/4249603701 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5805 (secure 0 0 0) 0x7f02bc099210 con 0x7f02cc0feb30 2023-07-14T19:23:21.913 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:21.913 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-14T19:23:21.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.913+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f02b80a11a0 msgr2=0x7f02b80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:21.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.913+0000 7f02d1de8700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f02b80a11a0 0x7f02b80a3650 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f02b4006010 tx=0x7f02b400b260 comp rx=0 tx=0).stop 2023-07-14T19:23:21.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.913+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02cc0feb30 msgr2=0x7f02cc069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:21.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.913+0000 7f02d1de8700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02cc0feb30 0x7f02cc069530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00d740 tx=0x7f02bc00da50 comp rx=0 tx=0).stop 2023-07-14T19:23:21.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.913+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 shutdown_connections 2023-07-14T19:23:21.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.913+0000 7f02d1de8700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f02b80a11a0 0x7f02b80a3650 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.913+0000 7f02d1de8700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02cc1006b0 0x7f02cc19cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.913+0000 7f02d1de8700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02cc0feb30 0x7f02cc069530 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.913+0000 7f02d1de8700 1 --2- 172.21.15.53:0/4249603701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f02cc0ffd20 0x7f02cc19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.914+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 >> 172.21.15.53:0/4249603701 conn(0x7f02cc0fa150 msgr2=0x7f02cc1031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:21.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.914+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 shutdown_connections 2023-07-14T19:23:21.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:21.914+0000 7f02d1de8700 1 -- 172.21.15.53:0/4249603701 wait complete. 2023-07-14T19:23:21.917 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 8 2023-07-14T19:23:21.980 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:21.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:21.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:21.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:21.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:21.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:21.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: Deploying daemon mds.h on smithi070 2023-07-14T19:23:21.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.104:6836/1255142015,v1:172.21.15.104:6837/1255142015] up:boot 2023-07-14T19:23:21.982 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: fsmap 7 up:standby 2023-07-14T19:23:21.982 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:23:21.982 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:21 smithi070 ceph-mon[98015]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:22.071 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:22.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:22.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:22.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:22.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:22.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:22.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: Deploying daemon mds.h on smithi070 2023-07-14T19:23:22.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.104:6836/1255142015,v1:172.21.15.104:6837/1255142015] up:boot 2023-07-14T19:23:22.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: fsmap 7 up:standby 2023-07-14T19:23:22.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:23:22.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:21 smithi053 ceph-mon[96565]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:22.260 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:22.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:22.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:22.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:22.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:22.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:22.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: Deploying daemon mds.h on smithi070 2023-07-14T19:23:22.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.104:6836/1255142015,v1:172.21.15.104:6837/1255142015] up:boot 2023-07-14T19:23:22.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: fsmap 7 up:standby 2023-07-14T19:23:22.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:23:22.262 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:21 smithi104 ceph-mon[98435]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:22.583 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:23:22 smithi070 systemd[1]: Starting Ceph mds.h for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:22.885 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:22 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/4249603701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:22.886 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:23:22 smithi070 podman[123058]: 2023-07-14T19:23:22.886 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:23:22 smithi070 ceph-mds[123098]: -- 172.21.15.70:0/3688113736 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55872c4e71e0 con 0x55872c4bd000 2023-07-14T19:23:22.886 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:23:22 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-h[123074]: starting mds.h at 2023-07-14T19:23:22.886 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:23:22 smithi070 bash[123058]: ceb92a0e25a0681003ad92d3db6d3251513046a3196e5494d7bcd295679947c3 2023-07-14T19:23:23.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:22 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/4249603701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:23.227 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:23:22 smithi070 systemd[1]: Started Ceph mds.h for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:23.261 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:22 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/4249603701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:23.928 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:24.070 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.068+0000 7f2f4529e700 1 -- 172.21.15.53:0/139268623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f400ffc10 msgr2=0x7f2f40100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:24.071 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.068+0000 7f2f4529e700 1 --2- 172.21.15.53:0/139268623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f400ffc10 0x7f2f40100020 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009990 tx=0x7f2f30009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.071 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.068+0000 7f2f4529e700 1 -- 172.21.15.53:0/139268623 shutdown_connections 2023-07-14T19:23:24.071 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.068+0000 7f2f4529e700 1 --2- 172.21.15.53:0/139268623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f40101790 0x7f2f40108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.071 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.068+0000 7f2f4529e700 1 --2- 172.21.15.53:0/139268623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f400ffc10 0x7f2f40100020 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.071 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.068+0000 7f2f4529e700 1 --2- 172.21.15.53:0/139268623 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40100e00 0x7f2f40101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.072 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.068+0000 7f2f4529e700 1 -- 172.21.15.53:0/139268623 >> 172.21.15.53:0/139268623 conn(0x7f2f400fb230 msgr2=0x7f2f400fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:24.072 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.068+0000 7f2f4529e700 1 -- 172.21.15.53:0/139268623 shutdown_connections 2023-07-14T19:23:24.072 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.068+0000 7f2f4529e700 1 -- 172.21.15.53:0/139268623 wait complete. 2023-07-14T19:23:24.072 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.069+0000 7f2f4529e700 1 Processor -- start 2023-07-14T19:23:24.072 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.069+0000 7f2f4529e700 1 -- start start 2023-07-14T19:23:24.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.069+0000 7f2f4529e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400ffc10 0x7f2f40195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:24.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.069+0000 7f2f4529e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f40100e00 0x7f2f40195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:24.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.069+0000 7f2f4529e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40101790 0x7f2f4019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:24.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.069+0000 7f2f4529e700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4019d490 con 0x7f2f40100e00 2023-07-14T19:23:24.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.069+0000 7f2f4529e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4019d5d0 con 0x7f2f400ffc10 2023-07-14T19:23:24.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.069+0000 7f2f4529e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4019d760 con 0x7f2f40101790 2023-07-14T19:23:24.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3e7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f40100e00 0x7f2f40195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:24.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3e7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f40100e00 0x7f2f40195e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50030/0 (socket says 172.21.15.53:50030) 2023-07-14T19:23:24.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3e7fc700 1 -- 172.21.15.53:0/1045304092 learned_addr learned my addr 172.21.15.53:0/1045304092 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:24.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3effd700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400ffc10 0x7f2f40195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:24.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3f7fe700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40101790 0x7f2f4019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:24.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3e7fc700 1 -- 172.21.15.53:0/1045304092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40101790 msgr2=0x7f2f4019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:24.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3e7fc700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40101790 0x7f2f4019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3e7fc700 1 -- 172.21.15.53:0/1045304092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400ffc10 msgr2=0x7f2f40195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:24.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3e7fc700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400ffc10 0x7f2f40195910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3e7fc700 1 -- 172.21.15.53:0/1045304092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300095f0 con 0x7f2f40100e00 2023-07-14T19:23:24.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f3e7fc700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f40100e00 0x7f2f40195e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400d950 tx=0x7f2f3400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:24.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f27fff700 1 -- 172.21.15.53:0/1045304092 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3400f890 con 0x7f2f40100e00 2023-07-14T19:23:24.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f27fff700 1 -- 172.21.15.53:0/1045304092 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f34010460 con 0x7f2f40100e00 2023-07-14T19:23:24.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4019da40 con 0x7f2f40100e00 2023-07-14T19:23:24.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f27fff700 1 -- 172.21.15.53:0/1045304092 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3400e610 con 0x7f2f40100e00 2023-07-14T19:23:24.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.070+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4019df50 con 0x7f2f40100e00 2023-07-14T19:23:24.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.071+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4010b430 con 0x7f2f40100e00 2023-07-14T19:23:24.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.072+0000 7f2f27fff700 1 -- 172.21.15.53:0/1045304092 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2f340108d0 con 0x7f2f40100e00 2023-07-14T19:23:24.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.073+0000 7f2f27fff700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f280a11a0 0x7f2f280a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:24.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.073+0000 7f2f3effd700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f280a11a0 0x7f2f280a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:24.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.073+0000 7f2f27fff700 1 -- 172.21.15.53:0/1045304092 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2f340cc250 con 0x7f2f40100e00 2023-07-14T19:23:24.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.074+0000 7f2f3effd700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f280a11a0 0x7f2f280a3650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000b5c0 tx=0x7f2f30005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:24.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.078+0000 7f2f27fff700 1 -- 172.21.15.53:0/1045304092 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2f34095e40 con 0x7f2f40100e00 2023-07-14T19:23:24.184 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:24.185 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:24.185 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:24.185 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:24.185 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:24.185 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:24.186 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: Deploying daemon mds.i on smithi104 2023-07-14T19:23:24.186 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:24.186 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] up:boot 2023-07-14T19:23:24.186 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: fsmap 8 up:standby 2023-07-14T19:23:24.186 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:23 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:23:24.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:24.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:24.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:24.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:24.228 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:24.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:24.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: Deploying daemon mds.i on smithi104 2023-07-14T19:23:24.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:24.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] up:boot 2023-07-14T19:23:24.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: fsmap 8 up:standby 2023-07-14T19:23:24.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:23 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:23:24.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:24.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:24.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:24.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:24.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:24.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:24.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: Deploying daemon mds.i on smithi104 2023-07-14T19:23:24.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:24.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] up:boot 2023-07-14T19:23:24.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: fsmap 8 up:standby 2023-07-14T19:23:24.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:23 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:23:24.325 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.324+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f4010b620 con 0x7f2f40100e00 2023-07-14T19:23:24.325 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.324+0000 7f2f27fff700 1 -- 172.21.15.53:0/1045304092 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6562 (secure 0 0 0) 0x7f2f34099460 con 0x7f2f40100e00 2023-07-14T19:23:24.326 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:24.326 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T19:23:24.327 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f280a11a0 msgr2=0x7f2f280a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:24.328 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f280a11a0 0x7f2f280a3650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000b5c0 tx=0x7f2f30005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.328 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f40100e00 msgr2=0x7f2f40195e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:24.328 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f40100e00 0x7f2f40195e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400d950 tx=0x7f2f3400dc60 comp rx=0 tx=0).stop 2023-07-14T19:23:24.329 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 shutdown_connections 2023-07-14T19:23:24.329 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f280a11a0 0x7f2f280a3650 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.329 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40101790 0x7f2f4019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.329 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400ffc10 0x7f2f40195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.329 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 --2- 172.21.15.53:0/1045304092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f40100e00 0x7f2f40195e50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.329 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 >> 172.21.15.53:0/1045304092 conn(0x7f2f400fb230 msgr2=0x7f2f400fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:24.329 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 shutdown_connections 2023-07-14T19:23:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:24.326+0000 7f2f4529e700 1 -- 172.21.15.53:0/1045304092 wait complete. 2023-07-14T19:23:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 9 2023-07-14T19:23:24.511 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:23:24 smithi104 systemd[1]: Starting Ceph mds.i for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:25.010 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:24 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1045304092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:25.011 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:23:24 smithi104 podman[124928]: 2023-07-14T19:23:25.227 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:24 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1045304092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:25.321 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:24 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1045304092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:25.510 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:23:25 smithi104 ceph-mds[124968]: -- 172.21.15.104:0/2922553785 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e49e57f1e0 con 0x55e49e555000 2023-07-14T19:23:25.511 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:23:25 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-i[124944]: starting mds.i at 2023-07-14T19:23:25.511 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:23:25 smithi104 bash[124928]: 7bdb0c377ab2991198308498452acbc43b7910c4c55369151ea8c31fc50c06a0 2023-07-14T19:23:25.511 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:23:25 smithi104 systemd[1]: Started Ceph mds.i for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:26.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:26.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.474+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2569170129 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0e8100680 msgr2=0x7fb0e8106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:26.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.474+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2569170129 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0e8100680 0x7fb0e8106f40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000b0a0 tx=0x7fb0e000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.474+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2569170129 shutdown_connections 2023-07-14T19:23:26.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.474+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2569170129 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0e8100680 0x7fb0e8106f40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.474+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2569170129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e80feb00 0x7fb0e80fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.474+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2569170129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e80ffcf0 0x7fb0e8100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.474+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2569170129 >> 172.21.15.53:0/2569170129 conn(0x7fb0e80fa100 msgr2=0x7fb0e80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:26.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.475+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2569170129 shutdown_connections 2023-07-14T19:23:26.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.475+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2569170129 wait complete. 2023-07-14T19:23:26.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:26.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:26.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:26.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:26.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:26.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:26.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: Deploying daemon mds.g on smithi053 2023-07-14T19:23:26.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:26.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.104:6838/1650908884,v1:172.21.15.104:6839/1650908884] up:boot 2023-07-14T19:23:26.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: fsmap 9 up:standby 2023-07-14T19:23:26.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:26 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:23:26.480 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.475+0000 7fb0efd8a700 1 Processor -- start 2023-07-14T19:23:26.480 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:26.480 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:26.480 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:26.480 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:26.481 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:26.481 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:26.481 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: Deploying daemon mds.g on smithi053 2023-07-14T19:23:26.481 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:26.481 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.104:6838/1650908884,v1:172.21.15.104:6839/1650908884] up:boot 2023-07-14T19:23:26.481 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: fsmap 9 up:standby 2023-07-14T19:23:26.481 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:26 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:23:26.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.475+0000 7fb0efd8a700 1 -- start start 2023-07-14T19:23:26.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0efd8a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e80feb00 0x7fb0e8068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0efd8a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e80ffcf0 0x7fb0e81984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0efd8a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0e8100680 0x7fb0e8198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0efd8a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8199310 con 0x7fb0e8100680 2023-07-14T19:23:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0efd8a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8199480 con 0x7fb0e80feb00 2023-07-14T19:23:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0efd8a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8199640 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0edb26700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e80feb00 0x7fb0e8068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0ed325700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e80ffcf0 0x7fb0e81984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0edb26700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e80feb00 0x7fb0e8068370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.53:36854/0 (socket says 172.21.15.53:36854) 2023-07-14T19:23:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0edb26700 1 -- 172.21.15.53:0/2307425643 learned_addr learned my addr 172.21.15.53:0/2307425643 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0ee327700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0e8100680 0x7fb0e8198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0ed325700 1 -- 172.21.15.53:0/2307425643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e80feb00 msgr2=0x7fb0e8068370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0ed325700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e80feb00 0x7fb0e8068370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0ed325700 1 -- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0e8100680 msgr2=0x7fb0e8198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0ed325700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0e8100680 0x7fb0e8198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0ed325700 1 -- 172.21.15.53:0/2307425643 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e0009d00 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.476+0000 7fb0edb26700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e80feb00 0x7fb0e8068370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:23:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.477+0000 7fb0ed325700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e80ffcf0 0x7fb0e81984f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800c970 tx=0x7fb0d800cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.477+0000 7fb0deffd700 1 -- 172.21.15.53:0/2307425643 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8007b00 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.477+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e8199920 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.477+0000 7fb0deffd700 1 -- 172.21.15.53:0/2307425643 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0d800ce60 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.477+0000 7fb0deffd700 1 -- 172.21.15.53:0/2307425643 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8005440 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.477+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e8199e80 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.478+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e819a8d0 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.479+0000 7fb0deffd700 1 -- 172.21.15.53:0/2307425643 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb0d80055a0 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.480+0000 7fb0deffd700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb0d40a11a0 0x7fb0d40a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:26.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.480+0000 7fb0edb26700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb0d40a11a0 0x7fb0d40a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:26.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.480+0000 7fb0deffd700 1 -- 172.21.15.53:0/2307425643 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb0d80cc2d0 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.480+0000 7fb0edb26700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb0d40a11a0 0x7fb0d40a3650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4007fd0 tx=0x7fb0e4007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.483+0000 7fb0deffd700 1 -- 172.21.15.53:0/2307425643 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb0d8095e60 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.510 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:26.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:26.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:26.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:26.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:26.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:26.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: Deploying daemon mds.g on smithi053 2023-07-14T19:23:26.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:26.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.104:6838/1650908884,v1:172.21.15.104:6839/1650908884] up:boot 2023-07-14T19:23:26.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: fsmap 9 up:standby 2023-07-14T19:23:26.512 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:26 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:23:26.767 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.765+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0e8105e60 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.768 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.767+0000 7fb0deffd700 1 -- 172.21.15.53:0/2307425643 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7324 (secure 0 0 0) 0x7fb0d8099480 con 0x7fb0e80ffcf0 2023-07-14T19:23:26.768 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:26.768 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T19:23:26.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb0d40a11a0 msgr2=0x7fb0d40a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:26.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb0d40a11a0 0x7fb0d40a3650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4007fd0 tx=0x7fb0e4007480 comp rx=0 tx=0).stop 2023-07-14T19:23:26.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e80ffcf0 msgr2=0x7fb0e81984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:26.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e80ffcf0 0x7fb0e81984f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800c970 tx=0x7fb0d800cc80 comp rx=0 tx=0).stop 2023-07-14T19:23:26.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 shutdown_connections 2023-07-14T19:23:26.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb0d40a11a0 0x7fb0d40a3650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0e8100680 0x7fb0e8198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e80feb00 0x7fb0e8068370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 --2- 172.21.15.53:0/2307425643 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0e80ffcf0 0x7fb0e81984f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 >> 172.21.15.53:0/2307425643 conn(0x7fb0e80fa100 msgr2=0x7fb0e8103110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:26.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 shutdown_connections 2023-07-14T19:23:26.772 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:26.769+0000 7fb0efd8a700 1 -- 172.21.15.53:0/2307425643 wait complete. 2023-07-14T19:23:26.772 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 10 2023-07-14T19:23:27.072 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:23:26 smithi053 systemd[1]: Starting Ceph mds.g for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:27.403 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:27 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2307425643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:27.403 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:23:27 smithi053 podman[130974]: 2023-07-14T19:23:27.403 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:23:27 smithi053 ceph-mds[131014]: -- 172.21.15.53:0/523762999 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555d4932d1e0 con 0x555d49302800 2023-07-14T19:23:27.403 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:23:27 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-g[130990]: starting mds.g at 2023-07-14T19:23:27.404 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:23:27 smithi053 bash[130974]: 5d6e73cf16d0af5222e15d4877e252e03ac0bc953668b11db1e0615b4b30c00a 2023-07-14T19:23:27.477 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:27 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2307425643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:27.511 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:27 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2307425643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:27.822 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:23:27 smithi053 systemd[1]: Started Ceph mds.g for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:28.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:28 smithi053 ceph-mon[96565]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:28.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:28.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:28.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:28.400 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:28.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:28.401 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:28 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:28.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:28 smithi070 ceph-mon[98015]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:28.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:28.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:28.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:28.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:28.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:28.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:28 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:28.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:28 smithi104 ceph-mon[98435]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:28.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:28.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:28.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:28.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:28.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:28.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:28 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:28.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:28.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.921+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/3721711315 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd0105930 msgr2=0x7fdfd0107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:28.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.921+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/3721711315 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd0105930 0x7fdfd0107d30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800b0a0 tx=0x7fdfc800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.921+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/3721711315 shutdown_connections 2023-07-14T19:23:28.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.921+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/3721711315 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd0105930 0x7fdfd0107d30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.921+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/3721711315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfd01006f0 0x7fdfd0102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.924 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.921+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/3721711315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfd0103010 0x7fdfd01053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.924 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.921+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/3721711315 >> 172.21.15.53:0/3721711315 conn(0x7fdfd00fa130 msgr2=0x7fdfd00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:28.924 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.921+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/3721711315 shutdown_connections 2023-07-14T19:23:28.924 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.921+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/3721711315 wait complete. 2023-07-14T19:23:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.922+0000 7fdfd5b0d700 1 Processor -- start 2023-07-14T19:23:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfd5b0d700 1 -- start start 2023-07-14T19:23:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfd5b0d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd01006f0 0x7fdfd0195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfd5b0d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfd0103010 0x7fdfd0195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:28.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfd5b0d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfd0105930 0x7fdfd019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:28.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfd5b0d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd019d510 con 0x7fdfd01006f0 2023-07-14T19:23:28.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfd5b0d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd019d650 con 0x7fdfd0103010 2023-07-14T19:23:28.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfd5b0d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd019d7e0 con 0x7fdfd0105930 2023-07-14T19:23:28.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfcf7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd01006f0 0x7fdfd0195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:28.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfcf7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd01006f0 0x7fdfd0195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50064/0 (socket says 172.21.15.53:50064) 2023-07-14T19:23:28.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfcf7fe700 1 -- 172.21.15.53:0/1570718028 learned_addr learned my addr 172.21.15.53:0/1570718028 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:28.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfceffd700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfd0103010 0x7fdfd0195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:28.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.923+0000 7fdfcffff700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfd0105930 0x7fdfd019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:28.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfcf7fe700 1 -- 172.21.15.53:0/1570718028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfd0105930 msgr2=0x7fdfd019cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:28.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfcf7fe700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfd0105930 0x7fdfd019cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfcf7fe700 1 -- 172.21.15.53:0/1570718028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfd0103010 msgr2=0x7fdfd0195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:28.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfcf7fe700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfd0103010 0x7fdfd0195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfcf7fe700 1 -- 172.21.15.53:0/1570718028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc8009d00 con 0x7fdfd01006f0 2023-07-14T19:23:28.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfcf7fe700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd01006f0 0x7fdfd0195990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000e9a0 tx=0x7fdfc000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:28.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfccff9700 1 -- 172.21.15.53:0/1570718028 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0009790 con 0x7fdfd01006f0 2023-07-14T19:23:28.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfccff9700 1 -- 172.21.15.53:0/1570718028 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfc0010430 con 0x7fdfd01006f0 2023-07-14T19:23:28.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfccff9700 1 -- 172.21.15.53:0/1570718028 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0018680 con 0x7fdfd01006f0 2023-07-14T19:23:28.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd019dac0 con 0x7fdfd01006f0 2023-07-14T19:23:28.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.924+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd019dfd0 con 0x7fdfd01006f0 2023-07-14T19:23:28.930 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.925+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd00fbca0 con 0x7fdfd01006f0 2023-07-14T19:23:28.931 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.926+0000 7fdfccff9700 1 -- 172.21.15.53:0/1570718028 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fdfc00105a0 con 0x7fdfd01006f0 2023-07-14T19:23:28.931 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.927+0000 7fdfccff9700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfb80a1220 0x7fdfb80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:28.931 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.927+0000 7fdfceffd700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfb80a1220 0x7fdfb80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:28.931 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.927+0000 7fdfccff9700 1 -- 172.21.15.53:0/1570718028 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdfc0014070 con 0x7fdfd01006f0 2023-07-14T19:23:28.931 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.928+0000 7fdfceffd700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfb80a1220 0x7fdfb80a36d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4005950 tx=0x7fdfc400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:28.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:28.931+0000 7fdfccff9700 1 -- 172.21.15.53:0/1570718028 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdfc0051780 con 0x7fdfd01006f0 2023-07-14T19:23:29.201 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.200+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfd002c8e0 con 0x7fdfd01006f0 2023-07-14T19:23:29.202 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.201+0000 7fdfccff9700 1 -- 172.21.15.53:0/1570718028 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8079 (secure 0 0 0) 0x7fdfd002c8e0 con 0x7fdfd01006f0 2023-07-14T19:23:29.203 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:29.203 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T19:23:29.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfb80a1220 msgr2=0x7fdfb80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:29.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfb80a1220 0x7fdfb80a36d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4005950 tx=0x7fdfc400b410 comp rx=0 tx=0).stop 2023-07-14T19:23:29.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd01006f0 msgr2=0x7fdfd0195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:29.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd01006f0 0x7fdfd0195990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000e9a0 tx=0x7fdfc000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:23:29.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 shutdown_connections 2023-07-14T19:23:29.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdfb80a1220 0x7fdfb80a36d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:29.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdfd0105930 0x7fdfd019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:29.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd01006f0 0x7fdfd0195990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:29.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 --2- 172.21.15.53:0/1570718028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfd0103010 0x7fdfd0195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:29.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 >> 172.21.15.53:0/1570718028 conn(0x7fdfd00fa130 msgr2=0x7fdfd0101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:29.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 shutdown_connections 2023-07-14T19:23:29.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:29.203+0000 7fdfd5b0d700 1 -- 172.21.15.53:0/1570718028 wait complete. 2023-07-14T19:23:29.207 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 11 2023-07-14T19:23:29.322 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:23:29 smithi053 systemd[1]: Starting Ceph mds.a for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:29.322 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:29 smithi053 ceph-mon[96565]: Deploying daemon mds.a on smithi053 2023-07-14T19:23:29.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:29 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.53:6838/122790216,v1:172.21.15.53:6839/122790216] up:boot 2023-07-14T19:23:29.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:29 smithi053 ceph-mon[96565]: fsmap 10 up:standby 2023-07-14T19:23:29.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:29 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:23:29.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:29 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1570718028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:29.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:29 smithi070 ceph-mon[98015]: Deploying daemon mds.a on smithi053 2023-07-14T19:23:29.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:29 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.53:6838/122790216,v1:172.21.15.53:6839/122790216] up:boot 2023-07-14T19:23:29.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:29 smithi070 ceph-mon[98015]: fsmap 10 up:standby 2023-07-14T19:23:29.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:29 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:23:29.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:29 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1570718028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:29.750 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:23:29 smithi053 podman[131359]: 2023-07-14T19:23:29.750 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:23:29 smithi053 bash[131359]: e3061997a51f76647b979109cab0e2031225b4825efba3f3cbfc271ee785b890 2023-07-14T19:23:29.750 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:23:29 smithi053 ceph-mds[131399]: -- 172.21.15.53:0/1211619291 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e781c2f1e0 con 0x55e781c05000 2023-07-14T19:23:29.751 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:23:29 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-a[131375]: starting mds.a at 2023-07-14T19:23:29.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:29 smithi104 ceph-mon[98435]: Deploying daemon mds.a on smithi053 2023-07-14T19:23:29.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:29 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.53:6838/122790216,v1:172.21.15.53:6839/122790216] up:boot 2023-07-14T19:23:29.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:29 smithi104 ceph-mon[98435]: fsmap 10 up:standby 2023-07-14T19:23:29.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:29 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:23:29.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:29 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1570718028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:30.071 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:23:29 smithi053 systemd[1]: Started Ceph mds.a for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:30 smithi053 ceph-mon[96565]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:30.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:30.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:30 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:30.625 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:30 smithi070 ceph-mon[98015]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:30.625 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:30.626 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:30.626 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:30.626 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:30.626 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:30.626 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:30 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:30.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:30 smithi104 ceph-mon[98435]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:30.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:30.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:30.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:30.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:23:30.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:23:30.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:30 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:31.219 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:23:31 smithi070 systemd[1]: Starting Ceph mds.b for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:23:31.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:31.367 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 -- 172.21.15.53:0/2668930660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1324100680 msgr2=0x7f1324106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:31.367 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 --2- 172.21.15.53:0/2668930660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1324100680 0x7f1324106f40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f131c00b0a0 tx=0x7f131c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.368 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 -- 172.21.15.53:0/2668930660 shutdown_connections 2023-07-14T19:23:31.368 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 --2- 172.21.15.53:0/2668930660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1324100680 0x7f1324106f40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.368 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 --2- 172.21.15.53:0/2668930660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13240feb00 0x7f13240fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.368 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 --2- 172.21.15.53:0/2668930660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13240ffcf0 0x7f1324100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.368 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 -- 172.21.15.53:0/2668930660 >> 172.21.15.53:0/2668930660 conn(0x7f13240fa100 msgr2=0x7f13240fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:31.368 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 -- 172.21.15.53:0/2668930660 shutdown_connections 2023-07-14T19:23:31.368 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 -- 172.21.15.53:0/2668930660 wait complete. 2023-07-14T19:23:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.366+0000 7f132a214700 1 Processor -- start 2023-07-14T19:23:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f132a214700 1 -- start start 2023-07-14T19:23:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f132a214700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240feb00 0x7f1324102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f132a214700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13240ffcf0 0x7f13241010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f132a214700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1324100680 0x7f1324101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f132a214700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1324101db0 con 0x7f13240feb00 2023-07-14T19:23:31.374 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f132a214700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1324101f20 con 0x7f13240ffcf0 2023-07-14T19:23:31.374 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f132a214700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132419fa50 con 0x7f1324100680 2023-07-14T19:23:31.374 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f13237fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240feb00 0x7f1324102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:31.375 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f13237fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240feb00 0x7f1324102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50090/0 (socket says 172.21.15.53:50090) 2023-07-14T19:23:31.375 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f1322ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13240ffcf0 0x7f13241010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:31.375 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f13237fe700 1 -- 172.21.15.53:0/2048364686 learned_addr learned my addr 172.21.15.53:0/2048364686 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:31.376 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f1323fff700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1324100680 0x7f1324101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:31.376 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f13237fe700 1 -- 172.21.15.53:0/2048364686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1324100680 msgr2=0x7f1324101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:31.376 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f13237fe700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1324100680 0x7f1324101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.377 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f13237fe700 1 -- 172.21.15.53:0/2048364686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13240ffcf0 msgr2=0x7f13241010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:31.377 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f13237fe700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13240ffcf0 0x7f13241010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.377 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f13237fe700 1 -- 172.21.15.53:0/2048364686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c009d00 con 0x7f13240feb00 2023-07-14T19:23:31.377 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f1323fff700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1324100680 0x7f1324101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:23:31.377 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f1322ffd700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13240ffcf0 0x7f13241010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:23:31.378 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f13237fe700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240feb00 0x7f1324102a90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f131400d980 tx=0x7f131400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:31.378 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.367+0000 7f1320ff9700 1 -- 172.21.15.53:0/2048364686 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314004020 con 0x7f13240feb00 2023-07-14T19:23:31.378 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.368+0000 7f1320ff9700 1 -- 172.21.15.53:0/2048364686 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1314009730 con 0x7f13240feb00 2023-07-14T19:23:31.378 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.368+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132419fd30 con 0x7f13240feb00 2023-07-14T19:23:31.378 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.368+0000 7f1320ff9700 1 -- 172.21.15.53:0/2048364686 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314003a80 con 0x7f13240feb00 2023-07-14T19:23:31.379 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.368+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13241a01d0 con 0x7f13240feb00 2023-07-14T19:23:31.379 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.368+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132410a2e0 con 0x7f13240feb00 2023-07-14T19:23:31.379 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.370+0000 7f1320ff9700 1 -- 172.21.15.53:0/2048364686 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f13140098a0 con 0x7f13240feb00 2023-07-14T19:23:31.380 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.370+0000 7f1320ff9700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f130c0a1220 0x7f130c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:31.380 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.370+0000 7f1320ff9700 1 -- 172.21.15.53:0/2048364686 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f13140cc420 con 0x7f13240feb00 2023-07-14T19:23:31.380 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.370+0000 7f1322ffd700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f130c0a1220 0x7f130c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:31.381 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.371+0000 7f1322ffd700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f130c0a1220 0x7f130c0a36d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1318011490 tx=0x7f1318009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:31.381 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.374+0000 7f1320ff9700 1 -- 172.21.15.53:0/2048364686 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1314096090 con 0x7f13240feb00 2023-07-14T19:23:31.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:31 smithi070 ceph-mon[98015]: Deploying daemon mds.b on smithi070 2023-07-14T19:23:31.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:31 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.53:6840/2523933697,v1:172.21.15.53:6841/2523933697] up:boot 2023-07-14T19:23:31.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:31 smithi070 ceph-mon[98015]: fsmap 11 up:standby 2023-07-14T19:23:31.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:31 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:23:31.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:31 smithi053 ceph-mon[96565]: Deploying daemon mds.b on smithi070 2023-07-14T19:23:31.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:31 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.53:6840/2523933697,v1:172.21.15.53:6841/2523933697] up:boot 2023-07-14T19:23:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:31 smithi053 ceph-mon[96565]: fsmap 11 up:standby 2023-07-14T19:23:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:31 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:23:31.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.616+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f132410a4d0 con 0x7f13240feb00 2023-07-14T19:23:31.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.617+0000 7f1320ff9700 1 -- 172.21.15.53:0/2048364686 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8837 (secure 0 0 0) 0x7f13140996b0 con 0x7f13240feb00 2023-07-14T19:23:31.618 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:31.618 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T19:23:31.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f130c0a1220 msgr2=0x7f130c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:31.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f130c0a1220 0x7f130c0a36d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1318011490 tx=0x7f1318009260 comp rx=0 tx=0).stop 2023-07-14T19:23:31.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240feb00 msgr2=0x7f1324102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:31.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240feb00 0x7f1324102a90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f131400d980 tx=0x7f131400dc90 comp rx=0 tx=0).stop 2023-07-14T19:23:31.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 shutdown_connections 2023-07-14T19:23:31.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f130c0a1220 0x7f130c0a36d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1324100680 0x7f1324101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13240feb00 0x7f1324102a90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 --2- 172.21.15.53:0/2048364686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13240ffcf0 0x7f13241010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 >> 172.21.15.53:0/2048364686 conn(0x7f13240fa100 msgr2=0x7f1324067820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:31.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 shutdown_connections 2023-07-14T19:23:31.622 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:31.619+0000 7f132a214700 1 -- 172.21.15.53:0/2048364686 wait complete. 2023-07-14T19:23:31.622 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 12 2023-07-14T19:23:31.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:31 smithi104 ceph-mon[98435]: Deploying daemon mds.b on smithi070 2023-07-14T19:23:31.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:31 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.53:6840/2523933697,v1:172.21.15.53:6841/2523933697] up:boot 2023-07-14T19:23:31.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:31 smithi104 ceph-mon[98435]: fsmap 11 up:standby 2023-07-14T19:23:31.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:31 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:23:31.776 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:23:31 smithi070 podman[123397]: 2023-07-14T19:23:31.777 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:23:31 smithi070 bash[123397]: bfdb97c1e189a70a4e8779cc7b430851fd89f009af98be002a48c6ef83c0497a 2023-07-14T19:23:32.227 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:23:31 smithi070 ceph-mds[123437]: -- 172.21.15.70:0/133361685 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55674d6c11e0 con 0x55674d697000 2023-07-14T19:23:32.228 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:23:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-b[123413]: starting mds.b at 2023-07-14T19:23:32.228 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:23:31 smithi070 systemd[1]: Started Ceph mds.b for c5147628-227a-11ee-9b33-001a4aab830c. 2023-07-14T19:23:32.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:32 smithi053 ceph-mon[96565]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:32 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2048364686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:32 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:32 smithi070 ceph-mon[98015]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:32 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2048364686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:32 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:32.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:32 smithi104 ceph-mon[98435]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:32.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:32 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2048364686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:32.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:32.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:32 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:33 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] up:boot 2023-07-14T19:23:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:33 smithi053 ceph-mon[96565]: fsmap 12 up:standby 2023-07-14T19:23:33.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:33 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:23:33.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:33 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] up:boot 2023-07-14T19:23:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:33 smithi070 ceph-mon[98015]: fsmap 12 up:standby 2023-07-14T19:23:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:33 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:23:33.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:33 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] up:boot 2023-07-14T19:23:33.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:33 smithi104 ceph-mon[98435]: fsmap 12 up:standby 2023-07-14T19:23:33.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:33 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:23:33.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.772+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3830176535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da8102fa0 msgr2=0x7f8da8105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:33.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.772+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3830176535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da8102fa0 0x7f8da8105380 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009990 tx=0x7f8d98009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:33.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.772+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3830176535 shutdown_connections 2023-07-14T19:23:33.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.772+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3830176535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da81058c0 0x7f8da8107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:33.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.772+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3830176535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8100680 0x7f8da8102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:33.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.772+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3830176535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da8102fa0 0x7f8da8105380 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:33.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.772+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3830176535 >> 172.21.15.53:0/3830176535 conn(0x7f8da80fa100 msgr2=0x7f8da80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:33.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.772+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3830176535 shutdown_connections 2023-07-14T19:23:33.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.772+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3830176535 wait complete. 2023-07-14T19:23:33.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dafe2b700 1 Processor -- start 2023-07-14T19:23:33.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dafe2b700 1 -- start start 2023-07-14T19:23:33.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dafe2b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da8100680 0x7f8da8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:33.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dafe2b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8102fa0 0x7f8da8198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:33.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dafe2b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da81058c0 0x7f8da819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:33.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dafe2b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da819f680 con 0x7f8da81058c0 2023-07-14T19:23:33.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dafe2b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da819f7c0 con 0x7f8da8100680 2023-07-14T19:23:33.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dafe2b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da819f950 con 0x7f8da8102fa0 2023-07-14T19:23:33.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dae3c8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da81058c0 0x7f8da819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:33.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dae3c8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da81058c0 0x7f8da819f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50108/0 (socket says 172.21.15.53:50108) 2023-07-14T19:23:33.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dad3c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8102fa0 0x7f8da8198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:33.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dadbc7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da8100680 0x7f8da8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:33.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.773+0000 7f8dae3c8700 1 -- 172.21.15.53:0/3228298652 learned_addr learned my addr 172.21.15.53:0/3228298652 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:33.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8dae3c8700 1 -- 172.21.15.53:0/3228298652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8102fa0 msgr2=0x7f8da8198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:33.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8dae3c8700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8102fa0 0x7f8da8198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:33.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8dae3c8700 1 -- 172.21.15.53:0/3228298652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da8100680 msgr2=0x7f8da8197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:33.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8dae3c8700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da8100680 0x7f8da8197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:33.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8dae3c8700 1 -- 172.21.15.53:0/3228298652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d980095f0 con 0x7f8da81058c0 2023-07-14T19:23:33.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8dae3c8700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da81058c0 0x7f8da819f0b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8da000ada0 tx=0x7f8da000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:33.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8d9effd700 1 -- 172.21.15.53:0/3228298652 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da000f070 con 0x7f8da81058c0 2023-07-14T19:23:33.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8d9effd700 1 -- 172.21.15.53:0/3228298652 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8da0007b30 con 0x7f8da81058c0 2023-07-14T19:23:33.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da819fc30 con 0x7f8da81058c0 2023-07-14T19:23:33.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8d9effd700 1 -- 172.21.15.53:0/3228298652 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da0005280 con 0x7f8da81058c0 2023-07-14T19:23:33.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.774+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da81a00f0 con 0x7f8da81058c0 2023-07-14T19:23:33.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.775+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da8107480 con 0x7f8da81058c0 2023-07-14T19:23:33.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.776+0000 7f8d9effd700 1 -- 172.21.15.53:0/3228298652 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8da000a090 con 0x7f8da81058c0 2023-07-14T19:23:33.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.777+0000 7f8d9effd700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8d940a11a0 0x7f8d940a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:33.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.777+0000 7f8dadbc7700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8d940a11a0 0x7f8d940a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:33.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.777+0000 7f8d9effd700 1 -- 172.21.15.53:0/3228298652 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8da00cbfe0 con 0x7f8da81058c0 2023-07-14T19:23:33.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.777+0000 7f8dadbc7700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8d940a11a0 0x7f8d940a3650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8da4005950 tx=0x7f8da400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:33.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:33.780+0000 7f8d9effd700 1 -- 172.21.15.53:0/3228298652 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8da0099670 con 0x7f8da81058c0 2023-07-14T19:23:34.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.031+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8da802c8e0 con 0x7f8da81058c0 2023-07-14T19:23:34.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.031+0000 7f8d9effd700 1 -- 172.21.15.53:0/3228298652 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9595 (secure 0 0 0) 0x7f8da0099200 con 0x7f8da81058c0 2023-07-14T19:23:34.034 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:34.034 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24461,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24527,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/2496341573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":2496341573},{"type":"v1","addr":"172.21.15.70:6839","nonce":2496341573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-14T19:23:34.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8d940a11a0 msgr2=0x7f8d940a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8d940a11a0 0x7f8d940a3650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8da4005950 tx=0x7f8da400b410 comp rx=0 tx=0).stop 2023-07-14T19:23:34.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da81058c0 msgr2=0x7f8da819f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da81058c0 0x7f8da819f0b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8da000ada0 tx=0x7f8da000c330 comp rx=0 tx=0).stop 2023-07-14T19:23:34.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 shutdown_connections 2023-07-14T19:23:34.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8d940a11a0 0x7f8d940a3650 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da81058c0 0x7f8da819f0b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da8100680 0x7f8da8197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 --2- 172.21.15.53:0/3228298652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8da8102fa0 0x7f8da8198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 >> 172.21.15.53:0/3228298652 conn(0x7f8da80fa100 msgr2=0x7f8da80fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:34.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 shutdown_connections 2023-07-14T19:23:34.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.034+0000 7f8dafe2b700 1 -- 172.21.15.53:0/3228298652 wait complete. 2023-07-14T19:23:34.039 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 13 2023-07-14T19:23:34.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:23:34.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.175+0000 7f2cc359b700 1 -- 172.21.15.53:0/2039031232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc0ff710 msgr2=0x7f2cbc106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.175+0000 7f2cc359b700 1 --2- 172.21.15.53:0/2039031232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc0ff710 0x7f2cbc106fe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400b0a0 tx=0x7f2cb400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.175+0000 7f2cc359b700 1 -- 172.21.15.53:0/2039031232 shutdown_connections 2023-07-14T19:23:34.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.175+0000 7f2cc359b700 1 --2- 172.21.15.53:0/2039031232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc0ff710 0x7f2cbc106fe0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.175+0000 7f2cc359b700 1 --2- 172.21.15.53:0/2039031232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cbc1000e0 0x7f2cbc1004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.175+0000 7f2cc359b700 1 --2- 172.21.15.53:0/2039031232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cbc0fed80 0x7f2cbc0ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.175+0000 7f2cc359b700 1 -- 172.21.15.53:0/2039031232 >> 172.21.15.53:0/2039031232 conn(0x7f2cbc0f98a0 msgr2=0x7f2cbc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:34.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.175+0000 7f2cc359b700 1 -- 172.21.15.53:0/2039031232 shutdown_connections 2023-07-14T19:23:34.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.175+0000 7f2cc359b700 1 -- 172.21.15.53:0/2039031232 wait complete. 2023-07-14T19:23:34.179 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc359b700 1 Processor -- start 2023-07-14T19:23:34.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc359b700 1 -- start start 2023-07-14T19:23:34.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc359b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cbc0fed80 0x7f2cbc194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc359b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cbc0ff710 0x7f2cbc194dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc359b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc1000e0 0x7f2cbc19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc359b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc19c410 con 0x7f2cbc1000e0 2023-07-14T19:23:34.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc359b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc19c550 con 0x7f2cbc0fed80 2023-07-14T19:23:34.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc359b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc19c6e0 con 0x7f2cbc0ff710 2023-07-14T19:23:34.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc1b38700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc1000e0 0x7f2cbc19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc1337700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cbc0fed80 0x7f2cbc194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc0b36700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cbc0ff710 0x7f2cbc194dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.176+0000 7f2cc1b38700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc1000e0 0x7f2cbc19be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50122/0 (socket says 172.21.15.53:50122) 2023-07-14T19:23:34.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc1b38700 1 -- 172.21.15.53:0/1619803075 learned_addr learned my addr 172.21.15.53:0/1619803075 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:34.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc0b36700 1 -- 172.21.15.53:0/1619803075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cbc0fed80 msgr2=0x7f2cbc194890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.182 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc0b36700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cbc0fed80 0x7f2cbc194890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.182 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc0b36700 1 -- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc1000e0 msgr2=0x7f2cbc19be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.182 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc0b36700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc1000e0 0x7f2cbc19be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.182 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc0b36700 1 -- 172.21.15.53:0/1619803075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb4009d00 con 0x7f2cbc0ff710 2023-07-14T19:23:34.182 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc1b38700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc1000e0 0x7f2cbc19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:23:34.182 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc1337700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cbc0fed80 0x7f2cbc194890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:23:34.182 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc0b36700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cbc0ff710 0x7f2cbc194dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00e9f0 tx=0x7f2cac00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:34.183 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cb27fc700 1 -- 172.21.15.53:0/1619803075 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac00c9c0 con 0x7f2cbc0ff710 2023-07-14T19:23:34.183 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cbc0fdcd0 con 0x7f2cbc0ff710 2023-07-14T19:23:34.183 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cb27fc700 1 -- 172.21.15.53:0/1619803075 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2cac010430 con 0x7f2cbc0ff710 2023-07-14T19:23:34.183 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cb27fc700 1 -- 172.21.15.53:0/1619803075 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac018680 con 0x7f2cbc0ff710 2023-07-14T19:23:34.183 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.177+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cbc0fe1e0 con 0x7f2cbc0ff710 2023-07-14T19:23:34.183 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.178+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cbc10a2e0 con 0x7f2cbc0ff710 2023-07-14T19:23:34.184 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.179+0000 7f2cb27fc700 1 -- 172.21.15.53:0/1619803075 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2cac010860 con 0x7f2cbc0ff710 2023-07-14T19:23:34.184 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.180+0000 7f2cb27fc700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2ca80a11a0 0x7f2ca80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.184 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.180+0000 7f2cb27fc700 1 -- 172.21.15.53:0/1619803075 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2cac014070 con 0x7f2cbc0ff710 2023-07-14T19:23:34.184 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.180+0000 7f2cc1337700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2ca80a11a0 0x7f2ca80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.185 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.181+0000 7f2cc1337700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2ca80a11a0 0x7f2ca80a3650 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc067560 tx=0x7f2cb800d5d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:34.185 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.183+0000 7f2cb27fc700 1 -- 172.21.15.53:0/1619803075 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2cac096130 con 0x7f2cbc0ff710 2023-07-14T19:23:34.402 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.401+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cbc10a4d0 con 0x7f2cbc0ff710 2023-07-14T19:23:34.404 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.402+0000 7f2cb27fc700 1 -- 172.21.15.53:0/1619803075 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18357 (secure 0 0 0) 0x7f2cac099750 con 0x7f2cbc0ff710 2023-07-14T19:23:34.404 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:34.405 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":76,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:22:56.998864+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:23:34.406 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2ca80a11a0 msgr2=0x7f2ca80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.406 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2ca80a11a0 0x7f2ca80a3650 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc067560 tx=0x7f2cb800d5d0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.406 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cbc0ff710 msgr2=0x7f2cbc194dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.407 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cbc0ff710 0x7f2cbc194dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00e9f0 tx=0x7f2cac00ed00 comp rx=0 tx=0).stop 2023-07-14T19:23:34.407 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 shutdown_connections 2023-07-14T19:23:34.407 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2ca80a11a0 0x7f2ca80a3650 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.407 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cbc1000e0 0x7f2cbc19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.407 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cbc0fed80 0x7f2cbc194890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.407 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 --2- 172.21.15.53:0/1619803075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2cbc0ff710 0x7f2cbc194dd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.408 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 >> 172.21.15.53:0/1619803075 conn(0x7f2cbc0f98a0 msgr2=0x7f2cbc0fb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:34.408 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 shutdown_connections 2023-07-14T19:23:34.408 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.405+0000 7f2cc359b700 1 -- 172.21.15.53:0/1619803075 wait complete. 2023-07-14T19:23:34.418 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:19:51.775369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:23:34.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:23:34.552 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 -- 172.21.15.53:0/1067797051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 msgr2=0x7f73a8100090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/1067797051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 0x7f73a8100090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7398009990 tx=0x7f7398009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 -- 172.21.15.53:0/1067797051 shutdown_connections 2023-07-14T19:23:34.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/1067797051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73a8101800 0x7f73a81080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/1067797051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 0x7f73a8100090 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/1067797051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73a8100e70 0x7f73a81012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 -- 172.21.15.53:0/1067797051 >> 172.21.15.53:0/1067797051 conn(0x7f73a80fb2a0 msgr2=0x7f73a80fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:34.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 -- 172.21.15.53:0/1067797051 shutdown_connections 2023-07-14T19:23:34.554 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 -- 172.21.15.53:0/1067797051 wait complete. 2023-07-14T19:23:34.554 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 Processor -- start 2023-07-14T19:23:34.554 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.551+0000 7f73ae5c6700 1 -- start start 2023-07-14T19:23:34.555 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73ae5c6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 0x7f73a81959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73ae5c6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73a8100e70 0x7f73a8195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73ae5c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73a8101800 0x7f73a819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73ae5c6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a819d560 con 0x7f73a80ffc80 2023-07-14T19:23:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73ae5c6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a819d6a0 con 0x7f73a8101800 2023-07-14T19:23:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73ae5c6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a819d830 con 0x7f73a8100e70 2023-07-14T19:23:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73a7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 0x7f73a81959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73a7fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 0x7f73a81959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50132/0 (socket says 172.21.15.53:50132) 2023-07-14T19:23:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73a7fff700 1 -- 172.21.15.53:0/485609177 learned_addr learned my addr 172.21.15.53:0/485609177 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73a77fe700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73a8100e70 0x7f73a8195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73acb63700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73a8101800 0x7f73a819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73a7fff700 1 -- 172.21.15.53:0/485609177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73a8100e70 msgr2=0x7f73a8195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73a7fff700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73a8100e70 0x7f73a8195f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73a7fff700 1 -- 172.21.15.53:0/485609177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73a8101800 msgr2=0x7f73a819cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73a7fff700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73a8101800 0x7f73a819cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73a7fff700 1 -- 172.21.15.53:0/485609177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73980095f0 con 0x7f73a80ffc80 2023-07-14T19:23:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.552+0000 7f73acb63700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73a8101800 0x7f73a819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:23:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.553+0000 7f73a7fff700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 0x7f73a81959e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7398000c00 tx=0x7f73980047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.553+0000 7f73a57fa700 1 -- 172.21.15.53:0/485609177 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739801d070 con 0x7f73a80ffc80 2023-07-14T19:23:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.553+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a819dab0 con 0x7f73a80ffc80 2023-07-14T19:23:34.559 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.553+0000 7f73a57fa700 1 -- 172.21.15.53:0/485609177 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73980040e0 con 0x7f73a80ffc80 2023-07-14T19:23:34.559 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.553+0000 7f73a57fa700 1 -- 172.21.15.53:0/485609177 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398022ea0 con 0x7f73a80ffc80 2023-07-14T19:23:34.559 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.553+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a819dfb0 con 0x7f73a80ffc80 2023-07-14T19:23:34.559 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.554+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a81072c0 con 0x7f73a80ffc80 2023-07-14T19:23:34.560 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.555+0000 7f73a57fa700 1 -- 172.21.15.53:0/485609177 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7398003c20 con 0x7f73a80ffc80 2023-07-14T19:23:34.560 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.555+0000 7f73a57fa700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f73900a1150 0x7f73900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.560 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.556+0000 7f73a77fe700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f73900a1150 0x7f73900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.560 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.556+0000 7f73a57fa700 1 -- 172.21.15.53:0/485609177 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f73980cdbb0 con 0x7f73a80ffc80 2023-07-14T19:23:34.560 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.556+0000 7f73a77fe700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f73900a1150 0x7f73900a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f739c005950 tx=0x7f739c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:34.561 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.559+0000 7f73a57fa700 1 -- 172.21.15.53:0/485609177 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7398097830 con 0x7f73a80ffc80 2023-07-14T19:23:34.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:34 smithi053 ceph-mon[96565]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:34.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:34 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3228298652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:34.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:34 smithi070 ceph-mon[98015]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:34.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:34 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3228298652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:34.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:34 smithi104 ceph-mon[98435]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:34.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:34 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3228298652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:34.761 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.759+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f73a802ca70 con 0x7f73a80ffc80 2023-07-14T19:23:34.761 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.760+0000 7f73a57fa700 1 -- 172.21.15.53:0/485609177 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f739809ae50 con 0x7f73a80ffc80 2023-07-14T19:23:34.762 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2023-07-14T19:23:34.763 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f73900a1150 msgr2=0x7f73900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.763 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f73900a1150 0x7f73900a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f739c005950 tx=0x7f739c00a300 comp rx=0 tx=0).stop 2023-07-14T19:23:34.764 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 msgr2=0x7f73a81959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.764 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 0x7f73a81959e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7398000c00 tx=0x7f73980047b0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.764 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 shutdown_connections 2023-07-14T19:23:34.764 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f73900a1150 0x7f73900a3600 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.764 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73a8101800 0x7f73a819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.764 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a80ffc80 0x7f73a81959e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.764 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 --2- 172.21.15.53:0/485609177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73a8100e70 0x7f73a8195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.765 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 >> 172.21.15.53:0/485609177 conn(0x7f73a80fb2a0 msgr2=0x7f73a8104400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:34.765 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 shutdown_connections 2023-07-14T19:23:34.765 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.762+0000 7f73ae5c6700 1 -- 172.21.15.53:0/485609177 wait complete. 2023-07-14T19:23:34.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-14T19:23:34.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.910+0000 7f435eee0700 1 -- 172.21.15.53:0/1201927945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43581017b0 msgr2=0x7f4358108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.910+0000 7f435eee0700 1 --2- 172.21.15.53:0/1201927945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43581017b0 0x7f4358108070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f435000b0a0 tx=0x7f435000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.910+0000 7f435eee0700 1 -- 172.21.15.53:0/1201927945 shutdown_connections 2023-07-14T19:23:34.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.910+0000 7f435eee0700 1 --2- 172.21.15.53:0/1201927945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43581017b0 0x7f4358108070 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.910+0000 7f435eee0700 1 --2- 172.21.15.53:0/1201927945 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43580ffc30 0x7f4358100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.910+0000 7f435eee0700 1 --2- 172.21.15.53:0/1201927945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4358100e20 0x7f4358101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.910+0000 7f435eee0700 1 -- 172.21.15.53:0/1201927945 >> 172.21.15.53:0/1201927945 conn(0x7f43580fb210 msgr2=0x7f43580fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:34.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 -- 172.21.15.53:0/1201927945 shutdown_connections 2023-07-14T19:23:34.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 -- 172.21.15.53:0/1201927945 wait complete. 2023-07-14T19:23:34.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 Processor -- start 2023-07-14T19:23:34.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 -- start start 2023-07-14T19:23:34.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43580ffc30 0x7f43580743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4358100e20 0x7f4358072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43581017b0 0x7f4358072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4358073710 con 0x7f43580ffc30 2023-07-14T19:23:34.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4358073880 con 0x7f43581017b0 2023-07-14T19:23:34.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.911+0000 7f435eee0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435819fa20 con 0x7f4358100e20 2023-07-14T19:23:34.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4357fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4358100e20 0x7f4358072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4357fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4358100e20 0x7f4358072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:50396/0 (socket says 172.21.15.53:50396) 2023-07-14T19:23:34.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4357fff700 1 -- 172.21.15.53:0/1724127320 learned_addr learned my addr 172.21.15.53:0/1724127320 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:34.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f435d47d700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43581017b0 0x7f4358072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f435cc7c700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43580ffc30 0x7f43580743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4357fff700 1 -- 172.21.15.53:0/1724127320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43581017b0 msgr2=0x7f4358072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4357fff700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43581017b0 0x7f4358072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4357fff700 1 -- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43580ffc30 msgr2=0x7f43580743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4357fff700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43580ffc30 0x7f43580743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4357fff700 1 -- 172.21.15.53:0/1724127320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4344009520 con 0x7f4358100e20 2023-07-14T19:23:34.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f435d47d700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43581017b0 0x7f4358072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:23:34.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4357fff700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4358100e20 0x7f4358072a30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f434400eb30 tx=0x7f434400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:34.918 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4355ffb700 1 -- 172.21.15.53:0/1724127320 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344009880 con 0x7f4358100e20 2023-07-14T19:23:34.918 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4350009d00 con 0x7f4358100e20 2023-07-14T19:23:34.919 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4355ffb700 1 -- 172.21.15.53:0/1724127320 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43440044e0 con 0x7f4358100e20 2023-07-14T19:23:34.919 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.912+0000 7f4355ffb700 1 -- 172.21.15.53:0/1724127320 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344003d80 con 0x7f4358100e20 2023-07-14T19:23:34.919 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.913+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43581a0080 con 0x7f4358100e20 2023-07-14T19:23:34.919 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.914+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4358107080 con 0x7f4358100e20 2023-07-14T19:23:34.919 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.914+0000 7f4355ffb700 1 -- 172.21.15.53:0/1724127320 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4344003f20 con 0x7f4358100e20 2023-07-14T19:23:34.919 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.915+0000 7f4355ffb700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f43480a11a0 0x7f43480a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.919 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.915+0000 7f4355ffb700 1 -- 172.21.15.53:0/1724127320 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4344014070 con 0x7f4358100e20 2023-07-14T19:23:34.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.915+0000 7f435cc7c700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f43480a11a0 0x7f43480a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.915+0000 7f435cc7c700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f43480a11a0 0x7f43480a3650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4358100c80 tx=0x7f434c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:34.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:34.919+0000 7f4355ffb700 1 -- 172.21.15.53:0/1724127320 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f43440965a0 con 0x7f4358100e20 2023-07-14T19:23:35.123 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.122+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f435802ca60 con 0x7f4358100e20 2023-07-14T19:23:35.124 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.122+0000 7f4355ffb700 1 -- 172.21.15.53:0/1724127320 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4344099bc0 con 0x7f4358100e20 2023-07-14T19:23:35.124 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:35.124 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2023-07-14T19:23:35.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2023-07-14T19:23:35.125 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2023-07-14T19:23:35.125 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2023-07-14T19:23:35.125 INFO:teuthology.orchestra.run.smithi053.stdout: } 2023-07-14T19:23:35.125 INFO:teuthology.orchestra.run.smithi053.stdout:] 2023-07-14T19:23:35.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f43480a11a0 msgr2=0x7f43480a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:35.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f43480a11a0 0x7f43480a3650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4358100c80 tx=0x7f434c009260 comp rx=0 tx=0).stop 2023-07-14T19:23:35.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4358100e20 msgr2=0x7f4358072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:35.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4358100e20 0x7f4358072a30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f434400eb30 tx=0x7f434400ee40 comp rx=0 tx=0).stop 2023-07-14T19:23:35.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 shutdown_connections 2023-07-14T19:23:35.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f43480a11a0 0x7f43480a3650 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43581017b0 0x7f4358072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43580ffc30 0x7f43580743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 --2- 172.21.15.53:0/1724127320 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4358100e20 0x7f4358072a30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 >> 172.21.15.53:0/1724127320 conn(0x7f43580fb210 msgr2=0x7f43581041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:35.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 shutdown_connections 2023-07-14T19:23:35.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.125+0000 7f435eee0700 1 -- 172.21.15.53:0/1724127320 wait complete. 2023-07-14T19:23:35.138 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-14T19:23:35.139 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-14T19:23:35.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.280+0000 7fec7512f700 1 -- 172.21.15.53:0/2433089280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec70100f50 msgr2=0x7fec701013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:35.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.280+0000 7fec7512f700 1 --2- 172.21.15.53:0/2433089280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec70100f50 0x7fec701013a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fec64009990 tx=0x7fec64009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.280+0000 7fec7512f700 1 -- 172.21.15.53:0/2433089280 shutdown_connections 2023-07-14T19:23:35.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.280+0000 7fec7512f700 1 --2- 172.21.15.53:0/2433089280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec701018e0 0x7fec701081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.280+0000 7fec7512f700 1 --2- 172.21.15.53:0/2433089280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec700fdee0 0x7fec700fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.280+0000 7fec7512f700 1 --2- 172.21.15.53:0/2433089280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec70100f50 0x7fec701013a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.280+0000 7fec7512f700 1 -- 172.21.15.53:0/2433089280 >> 172.21.15.53:0/2433089280 conn(0x7fec700f98a0 msgr2=0x7fec700fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:35.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.280+0000 7fec7512f700 1 -- 172.21.15.53:0/2433089280 shutdown_connections 2023-07-14T19:23:35.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.280+0000 7fec7512f700 1 -- 172.21.15.53:0/2433089280 wait complete. 2023-07-14T19:23:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.281+0000 7fec7512f700 1 Processor -- start 2023-07-14T19:23:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.281+0000 7fec7512f700 1 -- start start 2023-07-14T19:23:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.281+0000 7fec7512f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec700fdee0 0x7fec70195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.281+0000 7fec7512f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec70100f50 0x7fec70195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:35.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.281+0000 7fec7512f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec701018e0 0x7fec7019cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.281+0000 7fec7512f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7019d4e0 con 0x7fec701018e0 2023-07-14T19:23:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.281+0000 7fec7512f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7019d620 con 0x7fec70100f50 2023-07-14T19:23:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.281+0000 7fec7512f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7019d7b0 con 0x7fec700fdee0 2023-07-14T19:23:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec700fdee0 0x7fec70195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec70100f50 0x7fec70195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec700fdee0 0x7fec70195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:50408/0 (socket says 172.21.15.53:50408) 2023-07-14T19:23:35.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec70100f50 0x7fec70195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.53:36976/0 (socket says 172.21.15.53:36976) 2023-07-14T19:23:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6ed9d700 1 -- 172.21.15.53:0/1340078229 learned_addr learned my addr 172.21.15.53:0/1340078229 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6f59e700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec701018e0 0x7fec7019cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6ed9d700 1 -- 172.21.15.53:0/1340078229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec70100f50 msgr2=0x7fec70195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6ed9d700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec70100f50 0x7fec70195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6ed9d700 1 -- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec701018e0 msgr2=0x7fec7019cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6ed9d700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec701018e0 0x7fec7019cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6ed9d700 1 -- 172.21.15.53:0/1340078229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640095f0 con 0x7fec700fdee0 2023-07-14T19:23:35.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec6ed9d700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec700fdee0 0x7fec70195960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fec6000e970 tx=0x7fec6000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec57fff700 1 -- 172.21.15.53:0/1340078229 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60009790 con 0x7fec700fdee0 2023-07-14T19:23:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec57fff700 1 -- 172.21.15.53:0/1340078229 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec6000c7e0 con 0x7fec700fdee0 2023-07-14T19:23:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7019da90 con 0x7fec700fdee0 2023-07-14T19:23:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec57fff700 1 -- 172.21.15.53:0/1340078229 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec60003c70 con 0x7fec700fdee0 2023-07-14T19:23:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.282+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7019dfa0 con 0x7fec700fdee0 2023-07-14T19:23:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.284+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec70107340 con 0x7fec700fdee0 2023-07-14T19:23:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.285+0000 7fec57fff700 1 -- 172.21.15.53:0/1340078229 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fec6000cc50 con 0x7fec700fdee0 2023-07-14T19:23:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.285+0000 7fec57fff700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fec580a1150 0x7fec580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.285+0000 7fec57fff700 1 -- 172.21.15.53:0/1340078229 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fec60014070 con 0x7fec700fdee0 2023-07-14T19:23:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.285+0000 7fec6e59c700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fec580a1150 0x7fec580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.286+0000 7fec6e59c700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fec580a1150 0x7fec580a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec6400b2e0 tx=0x7fec64005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.289+0000 7fec57fff700 1 -- 172.21.15.53:0/1340078229 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fec60096210 con 0x7fec700fdee0 2023-07-14T19:23:35.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:35.498+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fec7002cb60 con 0x7fec700fdee0 2023-07-14T19:23:35.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:35 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1619803075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:35 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/485609177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:23:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:35 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:35 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1724127320' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:23:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:35 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1619803075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:35 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/485609177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:23:35.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:35 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:35 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1724127320' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:23:35.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:35 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1619803075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:35.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:35 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/485609177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:23:35.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:35 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:35.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:35 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1724127320' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:23:36.446 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.445+0000 7fec57fff700 1 -- 172.21.15.53:0/1340078229 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7fec60003dd0 con 0x7fec700fdee0 2023-07-14T19:23:36.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fec580a1150 msgr2=0x7fec580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:36.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fec580a1150 0x7fec580a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec6400b2e0 tx=0x7fec64005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec700fdee0 msgr2=0x7fec70195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:36.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec700fdee0 0x7fec70195960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fec6000e970 tx=0x7fec6000ec80 comp rx=0 tx=0).stop 2023-07-14T19:23:36.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 shutdown_connections 2023-07-14T19:23:36.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fec580a1150 0x7fec580a3600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec701018e0 0x7fec7019cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fec700fdee0 0x7fec70195960 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 --2- 172.21.15.53:0/1340078229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec70100f50 0x7fec70195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 >> 172.21.15.53:0/1340078229 conn(0x7fec700f98a0 msgr2=0x7fec70104480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:36.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 shutdown_connections 2023-07-14T19:23:36.455 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.450+0000 7fec7512f700 1 -- 172.21.15.53:0/1340078229 wait complete. 2023-07-14T19:23:36.455 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2023-07-14T19:23:36.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-14T19:23:36.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:36 smithi053 ceph-mon[96565]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:36 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1340078229' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:23:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:36 smithi053 ceph-mon[96565]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:23:36.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:36 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:36.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.604+0000 7f572b9a9700 1 -- 172.21.15.53:0/4167037214 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57240feb10 msgr2=0x7f57240fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:36.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.604+0000 7f572b9a9700 1 --2- 172.21.15.53:0/4167037214 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57240feb10 0x7f57240fef20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5720009990 tx=0x7f5720009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.604+0000 7f572b9a9700 1 -- 172.21.15.53:0/4167037214 shutdown_connections 2023-07-14T19:23:36.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.604+0000 7f572b9a9700 1 --2- 172.21.15.53:0/4167037214 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5724100690 0x7f5724106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.604+0000 7f572b9a9700 1 --2- 172.21.15.53:0/4167037214 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57240feb10 0x7f57240fef20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.604+0000 7f572b9a9700 1 --2- 172.21.15.53:0/4167037214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57240ffd00 0x7f5724100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.604+0000 7f572b9a9700 1 -- 172.21.15.53:0/4167037214 >> 172.21.15.53:0/4167037214 conn(0x7f57240fa130 msgr2=0x7f57240fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:36.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.604+0000 7f572b9a9700 1 -- 172.21.15.53:0/4167037214 shutdown_connections 2023-07-14T19:23:36.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.604+0000 7f572b9a9700 1 -- 172.21.15.53:0/4167037214 wait complete. 2023-07-14T19:23:36.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f572b9a9700 1 Processor -- start 2023-07-14T19:23:36.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f572b9a9700 1 -- start start 2023-07-14T19:23:36.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f572b9a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57240feb10 0x7f5724068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:36.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f572b9a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57240ffd00 0x7f572419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:36.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f572b9a9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5724100690 0x7f572419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:36.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f572b9a9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572419d440 con 0x7f5724100690 2023-07-14T19:23:36.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f572b9a9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572419d5b0 con 0x7f57240ffd00 2023-07-14T19:23:36.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f572b9a9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572419d770 con 0x7f57240feb10 2023-07-14T19:23:36.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f5729f46700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5724100690 0x7f572419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:36.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f5729f46700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5724100690 0x7f572419cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50176/0 (socket says 172.21.15.53:50176) 2023-07-14T19:23:36.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f5729f46700 1 -- 172.21.15.53:0/736538896 learned_addr learned my addr 172.21.15.53:0/736538896 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:36.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f5729745700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57240feb10 0x7f5724068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:36.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f5728f44700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57240ffd00 0x7f572419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:36.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f5729f46700 1 -- 172.21.15.53:0/736538896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57240feb10 msgr2=0x7f5724068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:36.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.605+0000 7f5729f46700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57240feb10 0x7f5724068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.606+0000 7f5729f46700 1 -- 172.21.15.53:0/736538896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57240ffd00 msgr2=0x7f572419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:36.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.606+0000 7f5729f46700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57240ffd00 0x7f572419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:36.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.606+0000 7f5729f46700 1 -- 172.21.15.53:0/736538896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57200095f0 con 0x7f5724100690 2023-07-14T19:23:36.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.606+0000 7f5729f46700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5724100690 0x7f572419cca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f571c007d70 tx=0x7f571c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:36.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.606+0000 7f571a7fc700 1 -- 172.21.15.53:0/736538896 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c00ad40 con 0x7f5724100690 2023-07-14T19:23:36.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.606+0000 7f571a7fc700 1 -- 172.21.15.53:0/736538896 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f571c0128e0 con 0x7f5724100690 2023-07-14T19:23:36.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.606+0000 7f571a7fc700 1 -- 172.21.15.53:0/736538896 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c0116b0 con 0x7f5724100690 2023-07-14T19:23:36.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.606+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572419da50 con 0x7f5724100690 2023-07-14T19:23:36.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.606+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572419df10 con 0x7f5724100690 2023-07-14T19:23:36.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.607+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572410a2e0 con 0x7f5724100690 2023-07-14T19:23:36.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.608+0000 7f571a7fc700 1 -- 172.21.15.53:0/736538896 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f571c012420 con 0x7f5724100690 2023-07-14T19:23:36.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.609+0000 7f571a7fc700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57100a11d0 0x7f57100a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:36.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.609+0000 7f571a7fc700 1 -- 172.21.15.53:0/736538896 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f571c0cccf0 con 0x7f5724100690 2023-07-14T19:23:36.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.609+0000 7f5729745700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57100a11d0 0x7f57100a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:36.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.609+0000 7f5729745700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57100a11d0 0x7f57100a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f572000b2e0 tx=0x7f5720005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:36.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.612+0000 7f571a7fc700 1 -- 172.21.15.53:0/736538896 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f571c0966d0 con 0x7f5724100690 2023-07-14T19:23:36.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:36 smithi070 ceph-mon[98015]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:36 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1340078229' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:23:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:36 smithi070 ceph-mon[98015]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:23:36.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:36 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:36.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:36 smithi104 ceph-mon[98435]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:36.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:36.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:36.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:36.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:36 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1340078229' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:23:36.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:36 smithi104 ceph-mon[98435]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:23:36.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:36 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:36.816 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:36.815+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f57241059c0 con 0x7f5724100690 2023-07-14T19:23:37.448 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.446+0000 7f571a7fc700 1 -- 172.21.15.53:0/736538896 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v78) v1 ==== 170+0+0 (secure 0 0 0) 0x7f571c023350 con 0x7f5724100690 2023-07-14T19:23:37.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.450+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57100a11d0 msgr2=0x7f57100a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:37.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.450+0000 7f572b9a9700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57100a11d0 0x7f57100a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f572000b2e0 tx=0x7f5720005960 comp rx=0 tx=0).stop 2023-07-14T19:23:37.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.450+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5724100690 msgr2=0x7f572419cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:37.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.450+0000 7f572b9a9700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5724100690 0x7f572419cca0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f571c007d70 tx=0x7f571c00a340 comp rx=0 tx=0).stop 2023-07-14T19:23:37.457 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.455+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 shutdown_connections 2023-07-14T19:23:37.457 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.455+0000 7f572b9a9700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57100a11d0 0x7f57100a3680 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.457 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.455+0000 7f572b9a9700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5724100690 0x7f572419cca0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.457 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.455+0000 7f572b9a9700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57240feb10 0x7f5724068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.457 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.455+0000 7f572b9a9700 1 --2- 172.21.15.53:0/736538896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57240ffd00 0x7f572419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.458 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.455+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 >> 172.21.15.53:0/736538896 conn(0x7f57240fa130 msgr2=0x7f57240fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:37.458 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.456+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 shutdown_connections 2023-07-14T19:23:37.460 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.457+0000 7f572b9a9700 1 -- 172.21.15.53:0/736538896 wait complete. 2023-07-14T19:23:37.464 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2023-07-14T19:23:37.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-14T19:23:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:37 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:37 smithi053 ceph-mon[96565]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:23:37.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:37 smithi053 ceph-mon[96565]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T19:23:37.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:37 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/736538896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:23:37.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.609+0000 7fdd60617700 1 -- 172.21.15.53:0/3016539862 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580ffd00 msgr2=0x7fdd58100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:37.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.609+0000 7fdd60617700 1 --2- 172.21.15.53:0/3016539862 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580ffd00 0x7fdd58100150 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdd48009990 tx=0x7fdd48009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.609+0000 7fdd60617700 1 -- 172.21.15.53:0/3016539862 shutdown_connections 2023-07-14T19:23:37.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.609+0000 7fdd60617700 1 --2- 172.21.15.53:0/3016539862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd58100690 0x7fdd58106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.609+0000 7fdd60617700 1 --2- 172.21.15.53:0/3016539862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd580feb10 0x7fdd580fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.609+0000 7fdd60617700 1 --2- 172.21.15.53:0/3016539862 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580ffd00 0x7fdd58100150 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.609+0000 7fdd60617700 1 -- 172.21.15.53:0/3016539862 >> 172.21.15.53:0/3016539862 conn(0x7fdd580fa130 msgr2=0x7fdd580fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.609+0000 7fdd60617700 1 -- 172.21.15.53:0/3016539862 shutdown_connections 2023-07-14T19:23:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.609+0000 7fdd60617700 1 -- 172.21.15.53:0/3016539862 wait complete. 2023-07-14T19:23:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd60617700 1 Processor -- start 2023-07-14T19:23:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd60617700 1 -- start start 2023-07-14T19:23:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd60617700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580feb10 0x7fdd58197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd60617700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd580ffd00 0x7fdd581980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd60617700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd58100690 0x7fdd5819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd60617700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5819f710 con 0x7fdd580feb10 2023-07-14T19:23:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd60617700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5819f850 con 0x7fdd580ffd00 2023-07-14T19:23:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd60617700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5819f9e0 con 0x7fdd58100690 2023-07-14T19:23:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5e3b3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580feb10 0x7fdd58197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5e3b3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580feb10 0x7fdd58197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50190/0 (socket says 172.21.15.53:50190) 2023-07-14T19:23:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5e3b3700 1 -- 172.21.15.53:0/3880586996 learned_addr learned my addr 172.21.15.53:0/3880586996 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5dbb2700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd580ffd00 0x7fdd581980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:37.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5ebb4700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd58100690 0x7fdd5819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:37.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5e3b3700 1 -- 172.21.15.53:0/3880586996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd58100690 msgr2=0x7fdd5819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:37.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5e3b3700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd58100690 0x7fdd5819f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5e3b3700 1 -- 172.21.15.53:0/3880586996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd580ffd00 msgr2=0x7fdd581980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:37.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5e3b3700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd580ffd00 0x7fdd581980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5e3b3700 1 -- 172.21.15.53:0/3880586996 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480095f0 con 0x7fdd580feb10 2023-07-14T19:23:37.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.610+0000 7fdd5e3b3700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580feb10 0x7fdd58197b90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400d980 tx=0x7fdd5400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:37.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.611+0000 7fdd4f7fe700 1 -- 172.21.15.53:0/3880586996 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54004020 con 0x7fdd580feb10 2023-07-14T19:23:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.611+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5819fcc0 con 0x7fdd580feb10 2023-07-14T19:23:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.611+0000 7fdd4f7fe700 1 -- 172.21.15.53:0/3880586996 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd5400f690 con 0x7fdd580feb10 2023-07-14T19:23:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.611+0000 7fdd4f7fe700 1 -- 172.21.15.53:0/3880586996 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54003a60 con 0x7fdd580feb10 2023-07-14T19:23:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.611+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd581a01d0 con 0x7fdd580feb10 2023-07-14T19:23:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.611+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5810a2e0 con 0x7fdd580feb10 2023-07-14T19:23:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.613+0000 7fdd4f7fe700 1 -- 172.21.15.53:0/3880586996 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fdd5400fb00 con 0x7fdd580feb10 2023-07-14T19:23:37.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.613+0000 7fdd4f7fe700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdd440a1150 0x7fdd440a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:37.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.614+0000 7fdd4f7fe700 1 -- 172.21.15.53:0/3880586996 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fdd540cc730 con 0x7fdd580feb10 2023-07-14T19:23:37.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.614+0000 7fdd5dbb2700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdd440a1150 0x7fdd440a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:37.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.614+0000 7fdd5dbb2700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdd440a1150 0x7fdd440a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800b2e0 tx=0x7fdd48005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:37.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.617+0000 7fdd4f7fe700 1 -- 172.21.15.53:0/3880586996 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdd54095f90 con 0x7fdd580feb10 2023-07-14T19:23:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:37 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:37 smithi070 ceph-mon[98015]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:23:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:37 smithi070 ceph-mon[98015]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T19:23:37.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:37 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/736538896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:23:37.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:37 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:37.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:37 smithi104 ceph-mon[98435]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:23:37.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:37 smithi104 ceph-mon[98435]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T19:23:37.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:37 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/736538896' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:23:37.894 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:37.892+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fdd5810a470 con 0x7fdd580feb10 2023-07-14T19:23:38.449 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.447+0000 7fdd4f7fe700 1 -- 172.21.15.53:0/3880586996 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fdd540995b0 con 0x7fdd580feb10 2023-07-14T19:23:38.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.450+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdd440a1150 msgr2=0x7fdd440a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.450+0000 7fdd60617700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdd440a1150 0x7fdd440a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800b2e0 tx=0x7fdd48005960 comp rx=0 tx=0).stop 2023-07-14T19:23:38.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.450+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580feb10 msgr2=0x7fdd58197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.450+0000 7fdd60617700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580feb10 0x7fdd58197b90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400d980 tx=0x7fdd5400dc90 comp rx=0 tx=0).stop 2023-07-14T19:23:38.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.451+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 shutdown_connections 2023-07-14T19:23:38.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.451+0000 7fdd60617700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdd440a1150 0x7fdd440a3600 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.451+0000 7fdd60617700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdd58100690 0x7fdd5819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.451+0000 7fdd60617700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd580feb10 0x7fdd58197b90 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400d980 tx=0x7fdd5400dc90 comp rx=0 tx=0).stop 2023-07-14T19:23:38.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.451+0000 7fdd60617700 1 --2- 172.21.15.53:0/3880586996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd580ffd00 0x7fdd581980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.451+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 >> 172.21.15.53:0/3880586996 conn(0x7fdd580fa130 msgr2=0x7fdd580677a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:38.455 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.452+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 shutdown_connections 2023-07-14T19:23:38.455 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.452+0000 7fdd60617700 1 -- 172.21.15.53:0/3880586996 wait complete. 2023-07-14T19:23:38.455 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 2 and data pool 3 2023-07-14T19:23:38.469 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:23:38.602 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.600+0000 7f6c055ab700 1 -- 172.21.15.53:0/3495055134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00101790 msgr2=0x7f6c00108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.602 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.600+0000 7f6c055ab700 1 --2- 172.21.15.53:0/3495055134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00101790 0x7f6c00108050 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800b0a0 tx=0x7f6bf800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.602 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.600+0000 7f6c055ab700 1 -- 172.21.15.53:0/3495055134 shutdown_connections 2023-07-14T19:23:38.603 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.600+0000 7f6c055ab700 1 --2- 172.21.15.53:0/3495055134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00101790 0x7f6c00108050 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.603 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.600+0000 7f6c055ab700 1 --2- 172.21.15.53:0/3495055134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c000ffc10 0x7f6c00100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.603 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.600+0000 7f6c055ab700 1 --2- 172.21.15.53:0/3495055134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c00100e00 0x7f6c00101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.603 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.600+0000 7f6c055ab700 1 -- 172.21.15.53:0/3495055134 >> 172.21.15.53:0/3495055134 conn(0x7f6c000fb230 msgr2=0x7f6c000fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:38.603 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.600+0000 7f6c055ab700 1 -- 172.21.15.53:0/3495055134 shutdown_connections 2023-07-14T19:23:38.603 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.600+0000 7f6c055ab700 1 -- 172.21.15.53:0/3495055134 wait complete. 2023-07-14T19:23:38.603 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6c055ab700 1 Processor -- start 2023-07-14T19:23:38.604 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6c055ab700 1 -- start start 2023-07-14T19:23:38.605 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6c055ab700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c000ffc10 0x7f6c000743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:38.605 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6c055ab700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00100e00 0x7f6c00072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:38.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6c055ab700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c00101790 0x7f6c00072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:38.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6c055ab700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c00073710 con 0x7f6c00100e00 2023-07-14T19:23:38.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6c055ab700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c00073880 con 0x7f6c00101790 2023-07-14T19:23:38.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6c055ab700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0019fa60 con 0x7f6c000ffc10 2023-07-14T19:23:38.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6bfe7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00100e00 0x7f6c00072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:38.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6bfe7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00100e00 0x7f6c00072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50210/0 (socket says 172.21.15.53:50210) 2023-07-14T19:23:38.606 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6bfe7fc700 1 -- 172.21.15.53:0/79458019 learned_addr learned my addr 172.21.15.53:0/79458019 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6bfeffd700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c000ffc10 0x7f6c000743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.601+0000 7f6bff7fe700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c00101790 0x7f6c00072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6bfe7fc700 1 -- 172.21.15.53:0/79458019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c000ffc10 msgr2=0x7f6c000743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6bfe7fc700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c000ffc10 0x7f6c000743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6bfe7fc700 1 -- 172.21.15.53:0/79458019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c00101790 msgr2=0x7f6c00072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6bfe7fc700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c00101790 0x7f6c00072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6bfe7fc700 1 -- 172.21.15.53:0/79458019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf8009d00 con 0x7f6c00100e00 2023-07-14T19:23:38.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6bfe7fc700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00100e00 0x7f6c00072a30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400d980 tx=0x7f6bf400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:38.608 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6be7fff700 1 -- 172.21.15.53:0/79458019 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4004220 con 0x7f6c00100e00 2023-07-14T19:23:38.608 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6be7fff700 1 -- 172.21.15.53:0/79458019 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6bf4007640 con 0x7f6c00100e00 2023-07-14T19:23:38.608 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0019fd40 con 0x7f6c00100e00 2023-07-14T19:23:38.608 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6be7fff700 1 -- 172.21.15.53:0/79458019 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4005690 con 0x7f6c00100e00 2023-07-14T19:23:38.608 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.602+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c001a01f0 con 0x7f6c00100e00 2023-07-14T19:23:38.608 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.603+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c00106a50 con 0x7f6c00100e00 2023-07-14T19:23:38.608 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.604+0000 7f6be7fff700 1 -- 172.21.15.53:0/79458019 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6bf4004750 con 0x7f6c00100e00 2023-07-14T19:23:38.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.605+0000 7f6be7fff700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6be80a11a0 0x7f6be80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:38.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.605+0000 7f6bfeffd700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6be80a11a0 0x7f6be80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:38.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.605+0000 7f6be7fff700 1 -- 172.21.15.53:0/79458019 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6bf40cca80 con 0x7f6c00100e00 2023-07-14T19:23:38.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.605+0000 7f6bfeffd700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6be80a11a0 0x7f6be80a3650 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6c00100c60 tx=0x7f6bf0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:38.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.609+0000 7f6be7fff700 1 -- 172.21.15.53:0/79458019 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6bf4096280 con 0x7f6c00100e00 2023-07-14T19:23:38.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:38 smithi070 ceph-mon[98015]: pgmap v245: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:38.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:38 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/736538896' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:23:38.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:38 smithi070 ceph-mon[98015]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:23:38.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:38 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3880586996' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:23:38.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:38 smithi070 ceph-mon[98015]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:23:38.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:38 smithi070 ceph-mon[98015]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:23:38.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:38 smithi104 ceph-mon[98435]: pgmap v245: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:38 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/736538896' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:23:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:38 smithi104 ceph-mon[98435]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:23:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:38 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3880586996' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:23:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:38 smithi104 ceph-mon[98435]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:23:38.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:38 smithi104 ceph-mon[98435]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:23:38.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:38 smithi053 ceph-mon[96565]: pgmap v245: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:23:38.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:38 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/736538896' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:23:38.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:38 smithi053 ceph-mon[96565]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:23:38.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:38 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3880586996' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:23:38.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:38 smithi053 ceph-mon[96565]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:23:38.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:38 smithi053 ceph-mon[96565]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:23:38.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.835+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c0002cab0 con 0x7f6c00100e00 2023-07-14T19:23:38.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.837+0000 7f6be7fff700 1 -- 172.21.15.53:0/79458019 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+22067 (secure 0 0 0) 0x7f6bf4078e60 con 0x7f6c00100e00 2023-07-14T19:23:38.839 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:38.839 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":79,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:23:37.894107+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:23:38.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.839+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6be80a11a0 msgr2=0x7f6be80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.839+0000 7f6c055ab700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6be80a11a0 0x7f6be80a3650 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6c00100c60 tx=0x7f6bf0009260 comp rx=0 tx=0).stop 2023-07-14T19:23:38.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.839+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00100e00 msgr2=0x7f6c00072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.839+0000 7f6c055ab700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00100e00 0x7f6c00072a30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400d980 tx=0x7f6bf400dc90 comp rx=0 tx=0).stop 2023-07-14T19:23:38.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.839+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 shutdown_connections 2023-07-14T19:23:38.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.839+0000 7f6c055ab700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6be80a11a0 0x7f6be80a3650 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.839+0000 7f6c055ab700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c00101790 0x7f6c00072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.839+0000 7f6c055ab700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c000ffc10 0x7f6c000743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.839+0000 7f6c055ab700 1 --2- 172.21.15.53:0/79458019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c00100e00 0x7f6c00072a30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.840+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 >> 172.21.15.53:0/79458019 conn(0x7f6c000fb230 msgr2=0x7f6c000fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:38.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.840+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 shutdown_connections 2023-07-14T19:23:38.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.840+0000 7f6c055ab700 1 -- 172.21.15.53:0/79458019 wait complete. 2023-07-14T19:23:38.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:23:38.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.979+0000 7fd1b5749700 1 -- 172.21.15.53:0/3785884649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b00ffc30 msgr2=0x7fd1b0100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.979+0000 7fd1b5749700 1 --2- 172.21.15.53:0/3785884649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b00ffc30 0x7fd1b0100040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009990 tx=0x7fd1a0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.979+0000 7fd1b5749700 1 -- 172.21.15.53:0/3785884649 shutdown_connections 2023-07-14T19:23:38.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.979+0000 7fd1b5749700 1 --2- 172.21.15.53:0/3785884649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b01017b0 0x7fd1b0108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.979+0000 7fd1b5749700 1 --2- 172.21.15.53:0/3785884649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b00ffc30 0x7fd1b0100040 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.979+0000 7fd1b5749700 1 --2- 172.21.15.53:0/3785884649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1b0100e20 0x7fd1b0101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.979+0000 7fd1b5749700 1 -- 172.21.15.53:0/3785884649 >> 172.21.15.53:0/3785884649 conn(0x7fd1b00fb230 msgr2=0x7fd1b00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:38.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.979+0000 7fd1b5749700 1 -- 172.21.15.53:0/3785884649 shutdown_connections 2023-07-14T19:23:38.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.979+0000 7fd1b5749700 1 -- 172.21.15.53:0/3785884649 wait complete. 2023-07-14T19:23:38.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.980+0000 7fd1b5749700 1 Processor -- start 2023-07-14T19:23:38.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.980+0000 7fd1b5749700 1 -- start start 2023-07-14T19:23:38.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.980+0000 7fd1b5749700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1b00ffc30 0x7fd1b0195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:38.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.980+0000 7fd1b5749700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b0100e20 0x7fd1b0195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:38.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.980+0000 7fd1b5749700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b01017b0 0x7fd1b019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:38.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.980+0000 7fd1b5749700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b019d4c0 con 0x7fd1b01017b0 2023-07-14T19:23:38.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.980+0000 7fd1b5749700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b019d600 con 0x7fd1b0100e20 2023-07-14T19:23:38.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.980+0000 7fd1b5749700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b019d790 con 0x7fd1b00ffc30 2023-07-14T19:23:38.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.980+0000 7fd1af7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b01017b0 0x7fd1b019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:38.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1af7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b01017b0 0x7fd1b019cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50224/0 (socket says 172.21.15.53:50224) 2023-07-14T19:23:38.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1aeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1b00ffc30 0x7fd1b0195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:38.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1af7fe700 1 -- 172.21.15.53:0/1956765896 learned_addr learned my addr 172.21.15.53:0/1956765896 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:38.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1a67fc700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b0100e20 0x7fd1b0195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:38.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1af7fe700 1 -- 172.21.15.53:0/1956765896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1b00ffc30 msgr2=0x7fd1b0195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1af7fe700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1b00ffc30 0x7fd1b0195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1af7fe700 1 -- 172.21.15.53:0/1956765896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b0100e20 msgr2=0x7fd1b0195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1af7fe700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b0100e20 0x7fd1b0195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1af7fe700 1 -- 172.21.15.53:0/1956765896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd198009520 con 0x7fd1b01017b0 2023-07-14T19:23:38.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1af7fe700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b01017b0 0x7fd1b019cef0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800ef30 tx=0x7fd1a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:38.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1acff9700 1 -- 172.21.15.53:0/1956765896 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8018070 con 0x7fd1b01017b0 2023-07-14T19:23:38.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a00095f0 con 0x7fd1b01017b0 2023-07-14T19:23:38.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1acff9700 1 -- 172.21.15.53:0/1956765896 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1a800f040 con 0x7fd1b01017b0 2023-07-14T19:23:38.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1acff9700 1 -- 172.21.15.53:0/1956765896 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a800cdb0 con 0x7fd1b01017b0 2023-07-14T19:23:38.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.981+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b019dd40 con 0x7fd1b01017b0 2023-07-14T19:23:38.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.982+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b0107180 con 0x7fd1b01017b0 2023-07-14T19:23:38.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.983+0000 7fd1acff9700 1 -- 172.21.15.53:0/1956765896 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd1a8003a00 con 0x7fd1b01017b0 2023-07-14T19:23:38.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.984+0000 7fd1acff9700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd19c0a11a0 0x7fd19c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:38.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.984+0000 7fd1aeffd700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd19c0a11a0 0x7fd19c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:38.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.984+0000 7fd1acff9700 1 -- 172.21.15.53:0/1956765896 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd1a80ccd60 con 0x7fd1b01017b0 2023-07-14T19:23:38.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.985+0000 7fd1aeffd700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd19c0a11a0 0x7fd19c0a3650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000b2e0 tx=0x7fd1a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:38.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:38.988+0000 7fd1acff9700 1 -- 172.21.15.53:0/1956765896 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd1a8096690 con 0x7fd1b01017b0 2023-07-14T19:23:39.194 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.192+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1b002cb30 con 0x7fd1b01017b0 2023-07-14T19:23:39.195 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.194+0000 7fd1acff9700 1 -- 172.21.15.53:0/1956765896 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+22067 (secure 0 0 0) 0x7fd1a8099cb0 con 0x7fd1b01017b0 2023-07-14T19:23:39.196 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:39.196 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":79,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:23:37.894107+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:23:39.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd19c0a11a0 msgr2=0x7fd19c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:39.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd19c0a11a0 0x7fd19c0a3650 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000b2e0 tx=0x7fd1a0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b01017b0 msgr2=0x7fd1b019cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:39.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b01017b0 0x7fd1b019cef0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800ef30 tx=0x7fd1a800c3a0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 shutdown_connections 2023-07-14T19:23:39.199 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd19c0a11a0 0x7fd19c0a3650 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.200 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1b01017b0 0x7fd1b019cef0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.200 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1b00ffc30 0x7fd1b0195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.201 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 --2- 172.21.15.53:0/1956765896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1b0100e20 0x7fd1b0195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.201 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 >> 172.21.15.53:0/1956765896 conn(0x7fd1b00fb230 msgr2=0x7fd1b01042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:39.201 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 shutdown_connections 2023-07-14T19:23:39.201 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.196+0000 7fd1b5749700 1 -- 172.21.15.53:0/1956765896 wait complete. 2023-07-14T19:23:39.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-14T19:23:39.345 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.341+0000 7fe308fb3700 1 -- 172.21.15.53:0/2675206657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304100730 msgr2=0x7fe304102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:39.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.341+0000 7fe308fb3700 1 --2- 172.21.15.53:0/2675206657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304100730 0x7fe304102b10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc00b0a0 tx=0x7fe2fc00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.341+0000 7fe308fb3700 1 -- 172.21.15.53:0/2675206657 shutdown_connections 2023-07-14T19:23:39.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.341+0000 7fe308fb3700 1 --2- 172.21.15.53:0/2675206657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe304105970 0x7fe304107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.341+0000 7fe308fb3700 1 --2- 172.21.15.53:0/2675206657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304100730 0x7fe304102b10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.347 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.341+0000 7fe308fb3700 1 --2- 172.21.15.53:0/2675206657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe304103050 0x7fe304105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.347 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.341+0000 7fe308fb3700 1 -- 172.21.15.53:0/2675206657 >> 172.21.15.53:0/2675206657 conn(0x7fe3040fa130 msgr2=0x7fe3040fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:39.347 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.341+0000 7fe308fb3700 1 -- 172.21.15.53:0/2675206657 shutdown_connections 2023-07-14T19:23:39.348 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.341+0000 7fe308fb3700 1 -- 172.21.15.53:0/2675206657 wait complete. 2023-07-14T19:23:39.348 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe308fb3700 1 Processor -- start 2023-07-14T19:23:39.348 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe308fb3700 1 -- start start 2023-07-14T19:23:39.348 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe308fb3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe304100730 0x7fe3041000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:39.349 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe308fb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe304103050 0x7fe3040fe6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:39.349 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe308fb3700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304105970 0x7fe3040fec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:39.349 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe308fb3700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040ff3c0 con 0x7fe304105970 2023-07-14T19:23:39.349 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe308fb3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3040ff530 con 0x7fe304100730 2023-07-14T19:23:39.350 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe308fb3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30419fac0 con 0x7fe304103050 2023-07-14T19:23:39.350 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe302d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304105970 0x7fe3040fec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:39.350 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe302d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304105970 0x7fe3040fec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50238/0 (socket says 172.21.15.53:50238) 2023-07-14T19:23:39.351 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe302d9d700 1 -- 172.21.15.53:0/1479882959 learned_addr learned my addr 172.21.15.53:0/1479882959 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:39.351 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe301d9b700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe304103050 0x7fe3040fe6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:39.351 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.342+0000 7fe30259c700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe304100730 0x7fe3041000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:39.352 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe302d9d700 1 -- 172.21.15.53:0/1479882959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe304103050 msgr2=0x7fe3040fe6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:39.352 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe302d9d700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe304103050 0x7fe3040fe6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.352 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe302d9d700 1 -- 172.21.15.53:0/1479882959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe304100730 msgr2=0x7fe3041000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:39.352 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe302d9d700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe304100730 0x7fe3041000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:39.353 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe302d9d700 1 -- 172.21.15.53:0/1479882959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc009d00 con 0x7fe304105970 2023-07-14T19:23:39.353 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe302d9d700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304105970 0x7fe3040fec20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400d980 tx=0x7fe2f400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:39.353 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe2fb7fe700 1 -- 172.21.15.53:0/1479882959 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4004a30 con 0x7fe304105970 2023-07-14T19:23:39.353 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe2fb7fe700 1 -- 172.21.15.53:0/1479882959 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2f40054e0 con 0x7fe304105970 2023-07-14T19:23:39.354 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30419fda0 con 0x7fe304105970 2023-07-14T19:23:39.354 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe2fb7fe700 1 -- 172.21.15.53:0/1479882959 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f400fbc0 con 0x7fe304105970 2023-07-14T19:23:39.354 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.343+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3041a0250 con 0x7fe304105970 2023-07-14T19:23:39.354 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.344+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3040fbca0 con 0x7fe304105970 2023-07-14T19:23:39.356 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.345+0000 7fe2fb7fe700 1 -- 172.21.15.53:0/1479882959 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe2f4004c20 con 0x7fe304105970 2023-07-14T19:23:39.356 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.345+0000 7fe2fb7fe700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe2f00a11d0 0x7fe2f00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:39.356 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.346+0000 7fe2fb7fe700 1 -- 172.21.15.53:0/1479882959 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe2f40cc570 con 0x7fe304105970 2023-07-14T19:23:39.357 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.346+0000 7fe30259c700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe2f00a11d0 0x7fe2f00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:39.357 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.346+0000 7fe30259c700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe2f00a11d0 0x7fe2f00a3680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009fd0 tx=0x7fe2fc00b7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:39.357 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.350+0000 7fe2fb7fe700 1 -- 172.21.15.53:0/1479882959 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe2f4095d70 con 0x7fe304105970 2023-07-14T19:23:39.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3880586996' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:23:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:23:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:23:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/79458019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:39.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1956765896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:39.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:23:39.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:39 smithi053 ceph-mon[96565]: pgmap v249: 97 pgs: 20 active+clean, 21 creating+peering, 56 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T19:23:39.603 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:39.602+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe30402c8e0 con 0x7fe304105970 2023-07-14T19:23:39.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3880586996' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:23:39.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:23:39.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:23:39.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:39.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:39.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/79458019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:39.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1956765896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:39.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:23:39.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:39 smithi070 ceph-mon[98015]: pgmap v249: 97 pgs: 20 active+clean, 21 creating+peering, 56 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T19:23:39.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3880586996' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:23:39.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:23:39.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:23:39.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:39.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:39.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:23:39.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/79458019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:39.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1956765896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:39.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:23:39.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:39 smithi104 ceph-mon[98435]: pgmap v249: 97 pgs: 20 active+clean, 21 creating+peering, 56 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T19:23:40.456 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.454+0000 7fe2fb7fe700 1 -- 172.21.15.53:0/1479882959 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe2f4099390 con 0x7fe304105970 2023-07-14T19:23:40.459 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe2f00a11d0 msgr2=0x7fe2f00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:40.459 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe2f00a11d0 0x7fe2f00a3680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009fd0 tx=0x7fe2fc00b7f0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.459 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304105970 msgr2=0x7fe3040fec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:40.460 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304105970 0x7fe3040fec20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe2f400d980 tx=0x7fe2f400dc90 comp rx=0 tx=0).stop 2023-07-14T19:23:40.460 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 shutdown_connections 2023-07-14T19:23:40.460 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe2f00a11d0 0x7fe2f00a3680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.460 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe304105970 0x7fe3040fec20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.460 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe304100730 0x7fe3041000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.461 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 --2- 172.21.15.53:0/1479882959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe304103050 0x7fe3040fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.461 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 >> 172.21.15.53:0/1479882959 conn(0x7fe3040fa130 msgr2=0x7fe304101410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:40.461 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 shutdown_connections 2023-07-14T19:23:40.461 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.458+0000 7fe308fb3700 1 -- 172.21.15.53:0/1479882959 wait complete. 2023-07-14T19:23:40.474 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2023-07-14T19:23:40.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-07-14T19:23:40.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.610+0000 7feeabe96700 1 -- 172.21.15.53:0/81153512 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4105980 msgr2=0x7feea4107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:40.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.610+0000 7feeabe96700 1 --2- 172.21.15.53:0/81153512 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4105980 0x7feea4107d60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00b0a0 tx=0x7fee9c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.610+0000 7feeabe96700 1 -- 172.21.15.53:0/81153512 shutdown_connections 2023-07-14T19:23:40.612 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.610+0000 7feeabe96700 1 --2- 172.21.15.53:0/81153512 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4105980 0x7feea4107d60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.610+0000 7feeabe96700 1 --2- 172.21.15.53:0/81153512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feea4100740 0x7feea4102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.610+0000 7feeabe96700 1 --2- 172.21.15.53:0/81153512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feea4103060 0x7feea4105440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.610+0000 7feeabe96700 1 -- 172.21.15.53:0/81153512 >> 172.21.15.53:0/81153512 conn(0x7feea40fa160 msgr2=0x7feea40fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:40.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.610+0000 7feeabe96700 1 -- 172.21.15.53:0/81153512 shutdown_connections 2023-07-14T19:23:40.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.610+0000 7feeabe96700 1 -- 172.21.15.53:0/81153512 wait complete. 2023-07-14T19:23:40.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feeabe96700 1 Processor -- start 2023-07-14T19:23:40.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feeabe96700 1 -- start start 2023-07-14T19:23:40.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feeabe96700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4100740 0x7feea41000b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:40.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feeabe96700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feea4103060 0x7feea40fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:40.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feeabe96700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feea4105980 0x7feea40fec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feeabe96700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea40ff3d0 con 0x7feea4100740 2023-07-14T19:23:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feeabe96700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea40ff540 con 0x7feea4105980 2023-07-14T19:23:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feeabe96700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea419fa70 con 0x7feea4103060 2023-07-14T19:23:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feea9c32700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4100740 0x7feea41000b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feea9c32700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4100740 0x7feea41000b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50248/0 (socket says 172.21.15.53:50248) 2023-07-14T19:23:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feea9c32700 1 -- 172.21.15.53:0/2155094302 learned_addr learned my addr 172.21.15.53:0/2155094302 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feea9431700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feea4103060 0x7feea40fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feeaa433700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feea4105980 0x7feea40fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feea9c32700 1 -- 172.21.15.53:0/2155094302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feea4103060 msgr2=0x7feea40fe6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feea9c32700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feea4103060 0x7feea40fe6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feea9c32700 1 -- 172.21.15.53:0/2155094302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feea4105980 msgr2=0x7feea40fec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feea9c32700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feea4105980 0x7feea40fec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.611+0000 7feea9c32700 1 -- 172.21.15.53:0/2155094302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c009d00 con 0x7feea4100740 2023-07-14T19:23:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.612+0000 7feea9c32700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4100740 0x7feea41000b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feea000c920 tx=0x7feea000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.612+0000 7fee9affd700 1 -- 172.21.15.53:0/2155094302 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0004a30 con 0x7feea4100740 2023-07-14T19:23:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.612+0000 7fee9affd700 1 -- 172.21.15.53:0/2155094302 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feea00054e0 con 0x7feea4100740 2023-07-14T19:23:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.612+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea419fd50 con 0x7feea4100740 2023-07-14T19:23:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.612+0000 7fee9affd700 1 -- 172.21.15.53:0/2155094302 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0007bd0 con 0x7feea4100740 2023-07-14T19:23:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.612+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea41a01b0 con 0x7feea4100740 2023-07-14T19:23:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.613+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea40fbcd0 con 0x7feea4100740 2023-07-14T19:23:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.614+0000 7fee9affd700 1 -- 172.21.15.53:0/2155094302 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7feea0007d50 con 0x7feea4100740 2023-07-14T19:23:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.614+0000 7fee9affd700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fee900a1150 0x7fee900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.615+0000 7fee9affd700 1 -- 172.21.15.53:0/2155094302 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7feea00cc700 con 0x7feea4100740 2023-07-14T19:23:40.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.615+0000 7feea9431700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fee900a1150 0x7fee900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:40.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.615+0000 7feea9431700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fee900a1150 0x7fee900a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fee94005950 tx=0x7fee9400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:40.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.618+0000 7fee9affd700 1 -- 172.21.15.53:0/2155094302 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7feea0095fa0 con 0x7feea4100740 2023-07-14T19:23:40.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:40 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] up:active 2023-07-14T19:23:40.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:40 smithi070 ceph-mon[98015]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:40.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:40 smithi070 ceph-mon[98015]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:40.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:40 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1479882959' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:23:40.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:40 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] up:active 2023-07-14T19:23:40.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:40 smithi104 ceph-mon[98435]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:40.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:40 smithi104 ceph-mon[98435]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:40.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:40 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1479882959' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:23:40.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:40 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] up:active 2023-07-14T19:23:40.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:40 smithi053 ceph-mon[96565]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:40.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:40 smithi053 ceph-mon[96565]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:40.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:40 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1479882959' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:23:40.864 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:40.863+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7feea402cb50 con 0x7feea4100740 2023-07-14T19:23:41.465 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.464+0000 7fee9affd700 1 -- 172.21.15.53:0/2155094302 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7feea001d020 con 0x7feea4100740 2023-07-14T19:23:41.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fee900a1150 msgr2=0x7fee900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:41.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fee900a1150 0x7fee900a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fee94005950 tx=0x7fee9400b410 comp rx=0 tx=0).stop 2023-07-14T19:23:41.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4100740 msgr2=0x7feea41000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:41.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4100740 0x7feea41000b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feea000c920 tx=0x7feea000cc30 comp rx=0 tx=0).stop 2023-07-14T19:23:41.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 shutdown_connections 2023-07-14T19:23:41.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fee900a1150 0x7fee900a3600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feea4105980 0x7feea40fec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feea4100740 0x7feea41000b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 --2- 172.21.15.53:0/2155094302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feea4103060 0x7feea40fe6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 >> 172.21.15.53:0/2155094302 conn(0x7feea40fa160 msgr2=0x7feea41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:41.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 shutdown_connections 2023-07-14T19:23:41.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.467+0000 7feeabe96700 1 -- 172.21.15.53:0/2155094302 wait complete. 2023-07-14T19:23:41.486 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-14T19:23:41.623 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.622+0000 7faedca2f700 1 -- 172.21.15.53:0/3451933970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed8100690 msgr2=0x7faed8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:41.624 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.622+0000 7faedca2f700 1 --2- 172.21.15.53:0/3451933970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed8100690 0x7faed8106f50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faed000b0a0 tx=0x7faed000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.624 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.622+0000 7faedca2f700 1 -- 172.21.15.53:0/3451933970 shutdown_connections 2023-07-14T19:23:41.624 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.622+0000 7faedca2f700 1 --2- 172.21.15.53:0/3451933970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed8100690 0x7faed8106f50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.624 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.622+0000 7faedca2f700 1 --2- 172.21.15.53:0/3451933970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed80feb10 0x7faed80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.624 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.622+0000 7faedca2f700 1 --2- 172.21.15.53:0/3451933970 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faed80ffd00 0x7faed8100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.622+0000 7faedca2f700 1 -- 172.21.15.53:0/3451933970 >> 172.21.15.53:0/3451933970 conn(0x7faed80fa130 msgr2=0x7faed80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.622+0000 7faedca2f700 1 -- 172.21.15.53:0/3451933970 shutdown_connections 2023-07-14T19:23:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.622+0000 7faedca2f700 1 -- 172.21.15.53:0/3451933970 wait complete. 2023-07-14T19:23:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faedca2f700 1 Processor -- start 2023-07-14T19:23:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faedca2f700 1 -- start start 2023-07-14T19:23:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faedca2f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed80feb10 0x7faed8102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faedca2f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faed80ffd00 0x7faed81010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faedca2f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed8100690 0x7faed8101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faedca2f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed8101dc0 con 0x7faed80feb10 2023-07-14T19:23:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faedca2f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed8101f30 con 0x7faed8100690 2023-07-14T19:23:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faedca2f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed819fa80 con 0x7faed80ffd00 2023-07-14T19:23:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed80feb10 0x7faed8102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed80feb10 0x7faed8102aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50262/0 (socket says 172.21.15.53:50262) 2023-07-14T19:23:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed659c700 1 -- 172.21.15.53:0/2285348309 learned_addr learned my addr 172.21.15.53:0/2285348309 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed5d9b700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faed80ffd00 0x7faed81010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed6d9d700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed8100690 0x7faed8101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed659c700 1 -- 172.21.15.53:0/2285348309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faed80ffd00 msgr2=0x7faed81010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed659c700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faed80ffd00 0x7faed81010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed659c700 1 -- 172.21.15.53:0/2285348309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed8100690 msgr2=0x7faed8101620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed659c700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed8100690 0x7faed8101620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed659c700 1 -- 172.21.15.53:0/2285348309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed0009d00 con 0x7faed80feb10 2023-07-14T19:23:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.623+0000 7faed6d9d700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed8100690 0x7faed8101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:23:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.624+0000 7faed659c700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed80feb10 0x7faed8102aa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faec800c990 tx=0x7faec800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.624+0000 7faec77fe700 1 -- 172.21.15.53:0/2285348309 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8007b00 con 0x7faed80feb10 2023-07-14T19:23:41.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.624+0000 7faec77fe700 1 -- 172.21.15.53:0/2285348309 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faec8007c60 con 0x7faed80feb10 2023-07-14T19:23:41.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.624+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed819fd60 con 0x7faed80feb10 2023-07-14T19:23:41.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.624+0000 7faec77fe700 1 -- 172.21.15.53:0/2285348309 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8018660 con 0x7faed80feb10 2023-07-14T19:23:41.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.624+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed81a01c0 con 0x7faed80feb10 2023-07-14T19:23:41.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.625+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed810a340 con 0x7faed80feb10 2023-07-14T19:23:41.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.626+0000 7faec77fe700 1 -- 172.21.15.53:0/2285348309 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faec8004020 con 0x7faed80feb10 2023-07-14T19:23:41.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.626+0000 7faec77fe700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faec00a1150 0x7faec00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:41.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.627+0000 7faed5d9b700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faec00a1150 0x7faec00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:41.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.627+0000 7faec77fe700 1 -- 172.21.15.53:0/2285348309 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7faec80cc920 con 0x7faed80feb10 2023-07-14T19:23:41.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.627+0000 7faed5d9b700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faec00a1150 0x7faec00a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faecc005950 tx=0x7faecc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:41.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.630+0000 7faec77fe700 1 -- 172.21.15.53:0/2285348309 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faec80961b0 con 0x7faed80feb10 2023-07-14T19:23:41.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:41 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1479882959' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:23:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:41 smithi070 ceph-mon[98015]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:41 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2155094302' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T19:23:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:41 smithi070 ceph-mon[98015]: pgmap v250: 97 pgs: 64 active+clean, 21 creating+peering, 12 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2023-07-14T19:23:41.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:41 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1479882959' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:23:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:41 smithi104 ceph-mon[98435]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:41 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2155094302' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T19:23:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:41 smithi104 ceph-mon[98435]: pgmap v250: 97 pgs: 64 active+clean, 21 creating+peering, 12 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2023-07-14T19:23:41.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:41 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1479882959' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:23:41.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:41 smithi053 ceph-mon[96565]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:41.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:41 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2155094302' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T19:23:41.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:41 smithi053 ceph-mon[96565]: pgmap v250: 97 pgs: 64 active+clean, 21 creating+peering, 12 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2023-07-14T19:23:41.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:41.881+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7faed81a0830 con 0x7faed80feb10 2023-07-14T19:23:42.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.282+0000 7faec77fe700 1 -- 172.21.15.53:0/2285348309 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7faec80997d0 con 0x7faed80feb10 2023-07-14T19:23:42.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faec00a1150 msgr2=0x7faec00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:42.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faec00a1150 0x7faec00a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faecc005950 tx=0x7faecc00b410 comp rx=0 tx=0).stop 2023-07-14T19:23:42.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed80feb10 msgr2=0x7faed8102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:42.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed80feb10 0x7faed8102aa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faec800c990 tx=0x7faec800cca0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 shutdown_connections 2023-07-14T19:23:42.286 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7faec00a1150 0x7faec00a3600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed8100690 0x7faed8101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed80feb10 0x7faed8102aa0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 --2- 172.21.15.53:0/2285348309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faed80ffd00 0x7faed81010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.284+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 >> 172.21.15.53:0/2285348309 conn(0x7faed80fa130 msgr2=0x7faed8067820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:42.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.285+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 shutdown_connections 2023-07-14T19:23:42.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.285+0000 7faedca2f700 1 -- 172.21.15.53:0/2285348309 wait complete. 2023-07-14T19:23:42.303 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-14T19:23:42.437 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.435+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/886445656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc40fdec0 msgr2=0x7f4fc40fe2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:42.437 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.435+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/886445656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc40fdec0 0x7f4fc40fe2d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009990 tx=0x7f4fb4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.438 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.435+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/886445656 shutdown_connections 2023-07-14T19:23:42.438 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.435+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/886445656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fc41018c0 0x7f4fc4108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.438 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.435+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/886445656 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc40fdec0 0x7f4fc40fe2d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.438 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.435+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/886445656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc4100f30 0x7f4fc4101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.435+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/886445656 >> 172.21.15.53:0/886445656 conn(0x7f4fc40f98a0 msgr2=0x7f4fc40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:42.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.435+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/886445656 shutdown_connections 2023-07-14T19:23:42.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.435+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/886445656 wait complete. 2023-07-14T19:23:42.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.436+0000 7f4fc9cb8700 1 Processor -- start 2023-07-14T19:23:42.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.436+0000 7f4fc9cb8700 1 -- start start 2023-07-14T19:23:42.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.436+0000 7f4fc9cb8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fc40fdec0 0x7f4fc4195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:42.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.436+0000 7f4fc9cb8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc4100f30 0x7f4fc4196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:42.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.436+0000 7f4fc9cb8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc41018c0 0x7f4fc419d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:42.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.436+0000 7f4fc9cb8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc419d690 con 0x7f4fc4100f30 2023-07-14T19:23:42.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.436+0000 7f4fc9cb8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc419d7d0 con 0x7f4fc40fdec0 2023-07-14T19:23:42.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.436+0000 7f4fc9cb8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc419d960 con 0x7f4fc41018c0 2023-07-14T19:23:42.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.436+0000 7f4fc2ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc4100f30 0x7f4fc4196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:42.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc2ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc4100f30 0x7f4fc4196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50278/0 (socket says 172.21.15.53:50278) 2023-07-14T19:23:42.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc2ffd700 1 -- 172.21.15.53:0/4196643412 learned_addr learned my addr 172.21.15.53:0/4196643412 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:42.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc3fff700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc41018c0 0x7f4fc419d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:42.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc37fe700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fc40fdec0 0x7f4fc4195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:42.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc2ffd700 1 -- 172.21.15.53:0/4196643412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc41018c0 msgr2=0x7f4fc419d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:42.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc2ffd700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc41018c0 0x7f4fc419d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc2ffd700 1 -- 172.21.15.53:0/4196643412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fc40fdec0 msgr2=0x7f4fc4195b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:42.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc2ffd700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fc40fdec0 0x7f4fc4195b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:42.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc2ffd700 1 -- 172.21.15.53:0/4196643412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb40095f0 con 0x7f4fc4100f30 2023-07-14T19:23:42.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc2ffd700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc4100f30 0x7f4fc4196050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800b530 tx=0x7f4fb800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:42.444 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc0ff9700 1 -- 172.21.15.53:0/4196643412 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8011830 con 0x7f4fc4100f30 2023-07-14T19:23:42.444 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc0ff9700 1 -- 172.21.15.53:0/4196643412 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fb8011e50 con 0x7f4fc4100f30 2023-07-14T19:23:42.444 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc419dc40 con 0x7f4fc4100f30 2023-07-14T19:23:42.444 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc0ff9700 1 -- 172.21.15.53:0/4196643412 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb800f2e0 con 0x7f4fc4100f30 2023-07-14T19:23:42.444 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.437+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc419e1a0 con 0x7f4fc4100f30 2023-07-14T19:23:42.444 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.438+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc410b520 con 0x7f4fc4100f30 2023-07-14T19:23:42.444 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.439+0000 7f4fc0ff9700 1 -- 172.21.15.53:0/4196643412 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4fb8010580 con 0x7f4fc4100f30 2023-07-14T19:23:42.445 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.440+0000 7f4fc0ff9700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4fac0a11d0 0x7f4fac0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:42.445 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.440+0000 7f4fc0ff9700 1 -- 172.21.15.53:0/4196643412 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4fb80cc7c0 con 0x7f4fc4100f30 2023-07-14T19:23:42.445 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.440+0000 7f4fc37fe700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4fac0a11d0 0x7f4fac0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:42.445 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.440+0000 7f4fc37fe700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4fac0a11d0 0x7f4fac0a3680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4000c00 tx=0x7f4fb4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:42.446 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.443+0000 7f4fc0ff9700 1 -- 172.21.15.53:0/4196643412 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4fb8096060 con 0x7f4fc4100f30 2023-07-14T19:23:42.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:23:42.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2155094302' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T19:23:42.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:23:42.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:42.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-07-14T19:23:42.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-07-14T19:23:42.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2285348309' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:23:42.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] up:active 2023-07-14T19:23:42.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2285348309' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:23:42.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:23:42.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:23:42.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: Cluster is now healthy 2023-07-14T19:23:42.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2023-07-14T19:23:42.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 2 up:standby-replay 7 up:standby 2023-07-14T19:23:42.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:42 smithi053 ceph-mon[96565]: daemon mds.e is now active in filesystem cephfs as rank 2 2023-07-14T19:23:42.695 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:42.694+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f4fc410b740 con 0x7f4fc4100f30 2023-07-14T19:23:42.727 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-b[123413]: 2023-07-14T19:23:42.298+0000 7f1580312700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-14T19:23:42.728 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-b[123413]: 2023-07-14T19:23:42.298+0000 7f1580312700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-14T19:23:42.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:23:42.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2155094302' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T19:23:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:23:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-07-14T19:23:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-07-14T19:23:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2285348309' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:23:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] up:active 2023-07-14T19:23:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2285348309' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:23:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:23:42.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:23:42.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: Cluster is now healthy 2023-07-14T19:23:42.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2023-07-14T19:23:42.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 2 up:standby-replay 7 up:standby 2023-07-14T19:23:42.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:42 smithi070 ceph-mon[98015]: daemon mds.e is now active in filesystem cephfs as rank 2 2023-07-14T19:23:42.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:23:42.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2155094302' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T19:23:42.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:23:42.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:23:42.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-07-14T19:23:42.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-07-14T19:23:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2285348309' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:23:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] up:active 2023-07-14T19:23:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2285348309' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:23:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:23:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:23:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: Cluster is now healthy 2023-07-14T19:23:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2023-07-14T19:23:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 2 up:standby-replay 7 up:standby 2023-07-14T19:23:42.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:42 smithi104 ceph-mon[98435]: daemon mds.e is now active in filesystem cephfs as rank 2 2023-07-14T19:23:43.289 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.288+0000 7f4fc0ff9700 1 -- 172.21.15.53:0/4196643412 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4fb8015020 con 0x7f4fc4100f30 2023-07-14T19:23:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4fac0a11d0 msgr2=0x7f4fac0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4fac0a11d0 0x7f4fac0a3680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4000c00 tx=0x7f4fb4005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc4100f30 msgr2=0x7f4fc4196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc4100f30 0x7f4fc4196050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4fb800b530 tx=0x7f4fb800b8f0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 shutdown_connections 2023-07-14T19:23:43.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4fac0a11d0 0x7f4fac0a3680 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4fc41018c0 0x7f4fc419d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fc40fdec0 0x7f4fc4195b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 --2- 172.21.15.53:0/4196643412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4fc4100f30 0x7f4fc4196050 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.291+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 >> 172.21.15.53:0/4196643412 conn(0x7f4fc40f98a0 msgr2=0x7f4fc40fbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.292+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 shutdown_connections 2023-07-14T19:23:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.292+0000 7f4fc9cb8700 1 -- 172.21.15.53:0/4196643412 wait complete. 2023-07-14T19:23:43.309 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2023-07-14T19:23:43.309 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-07-14T19:23:43.445 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/3255754068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fe500 msgr2=0x7f4ac00fe910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.446 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/3255754068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fe500 0x7f4ac00fe910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009990 tx=0x7f4ab4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.446 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/3255754068 shutdown_connections 2023-07-14T19:23:43.446 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/3255754068 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ac00ff800 0x7f4ac00ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/3255754068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fe500 0x7f4ac00fe910 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/3255754068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ac00fee50 0x7f4ac00ff2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/3255754068 >> 172.21.15.53:0/3255754068 conn(0x7f4ac00fa130 msgr2=0x7f4ac00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/3255754068 shutdown_connections 2023-07-14T19:23:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/3255754068 wait complete. 2023-07-14T19:23:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 Processor -- start 2023-07-14T19:23:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 -- start start 2023-07-14T19:23:43.450 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.444+0000 7f4ac6a7b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ac00fe500 0x7f4ac0197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:43.450 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4ac6a7b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fee50 0x7f4ac0198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:43.450 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4ac6a7b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ac00ff800 0x7f4ac019f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:43.451 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4ac6a7b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac019f880 con 0x7f4ac00fee50 2023-07-14T19:23:43.451 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4ac6a7b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac019f9f0 con 0x7f4ac00ff800 2023-07-14T19:23:43.451 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4ac6a7b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac019fbb0 con 0x7f4ac00fe500 2023-07-14T19:23:43.451 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fee50 0x7f4ac0198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:43.451 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fee50 0x7f4ac0198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50292/0 (socket says 172.21.15.53:50292) 2023-07-14T19:23:43.451 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abffff700 1 -- 172.21.15.53:0/584360702 learned_addr learned my addr 172.21.15.53:0/584360702 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:43.451 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4ac5018700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ac00ff800 0x7f4ac019f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:43.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4ac4817700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ac00fe500 0x7f4ac0197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:43.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abffff700 1 -- 172.21.15.53:0/584360702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ac00fe500 msgr2=0x7f4ac0197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abffff700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ac00fe500 0x7f4ac0197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abffff700 1 -- 172.21.15.53:0/584360702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ac00ff800 msgr2=0x7f4ac019f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abffff700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ac00ff800 0x7f4ac019f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abffff700 1 -- 172.21.15.53:0/584360702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab40095f0 con 0x7f4ac00fee50 2023-07-14T19:23:43.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abffff700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fee50 0x7f4ac0198280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00b530 tx=0x7f4aac00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:43.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abdffb700 1 -- 172.21.15.53:0/584360702 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac011830 con 0x7f4ac00fee50 2023-07-14T19:23:43.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abdffb700 1 -- 172.21.15.53:0/584360702 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4aac011e50 con 0x7f4ac00fee50 2023-07-14T19:23:43.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac019fe90 con 0x7f4ac00fee50 2023-07-14T19:23:43.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.445+0000 7f4abdffb700 1 -- 172.21.15.53:0/584360702 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac00f2e0 con 0x7f4ac00fee50 2023-07-14T19:23:43.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.446+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac01a0330 con 0x7f4ac00fee50 2023-07-14T19:23:43.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.447+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac01092e0 con 0x7f4ac00fee50 2023-07-14T19:23:43.453 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.448+0000 7f4abdffb700 1 -- 172.21.15.53:0/584360702 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4aac010380 con 0x7f4ac00fee50 2023-07-14T19:23:43.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.448+0000 7f4abdffb700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4ab00a1150 0x7f4ab00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:43.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.448+0000 7f4ac4817700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4ab00a1150 0x7f4ab00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:43.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.448+0000 7f4abdffb700 1 -- 172.21.15.53:0/584360702 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4aac0cc2e0 con 0x7f4ac00fee50 2023-07-14T19:23:43.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.449+0000 7f4ac4817700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4ab00a1150 0x7f4ab00a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400b5c0 tx=0x7f4ab4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:43.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.452+0000 7f4abdffb700 1 -- 172.21.15.53:0/584360702 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4aac095b80 con 0x7f4ac00fee50 2023-07-14T19:23:43.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/4196643412' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:23:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] up:active 2023-07-14T19:23:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/4196643412' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:23:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-mon[96565]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2023-07-14T19:23:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-mon[96565]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:23:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-mon[96565]: pgmap v251: 97 pgs: 97 active+clean; 580 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 2.8 KiB/s wr, 10 op/s 2023-07-14T19:23:43.689 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.688+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ac0060a80 con 0x7f4ab00a1150 2023-07-14T19:23:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:43 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/4196643412' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:23:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:43 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] up:active 2023-07-14T19:23:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:43 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/4196643412' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:23:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:43 smithi070 ceph-mon[98015]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2023-07-14T19:23:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:43 smithi070 ceph-mon[98015]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:23:43.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:43 smithi070 ceph-mon[98015]: pgmap v251: 97 pgs: 97 active+clean; 580 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 2.8 KiB/s wr, 10 op/s 2023-07-14T19:23:43.733 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.732+0000 7f4abdffb700 1 -- 172.21.15.53:0/584360702 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4ac0060a80 con 0x7f4ab00a1150 2023-07-14T19:23:43.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4ab00a1150 msgr2=0x7f4ab00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4ab00a1150 0x7f4ab00a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400b5c0 tx=0x7f4ab4005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fee50 msgr2=0x7f4ac0198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fee50 0x7f4ac0198280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00b530 tx=0x7f4aac00b840 comp rx=0 tx=0).stop 2023-07-14T19:23:43.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 shutdown_connections 2023-07-14T19:23:43.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4ab00a1150 0x7f4ab00a3600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ac00ff800 0x7f4ac019f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.737 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ac00fe500 0x7f4ac0197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.737 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 --2- 172.21.15.53:0/584360702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac00fee50 0x7f4ac0198280 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.737 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 >> 172.21.15.53:0/584360702 conn(0x7f4ac00fa130 msgr2=0x7f4ac00fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:43.737 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 shutdown_connections 2023-07-14T19:23:43.737 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.734+0000 7f4ac6a7b700 1 -- 172.21.15.53:0/584360702 wait complete. 2023-07-14T19:23:43.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-07-14T19:23:43.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:43 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/4196643412' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:23:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:43 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] up:active 2023-07-14T19:23:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:43 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/4196643412' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:23:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:43 smithi104 ceph-mon[98435]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2023-07-14T19:23:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:43 smithi104 ceph-mon[98435]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:23:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:43 smithi104 ceph-mon[98435]: pgmap v251: 97 pgs: 97 active+clean; 580 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 2.8 KiB/s wr, 10 op/s 2023-07-14T19:23:43.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.875+0000 7f3bc421d700 1 -- 172.21.15.53:0/915438632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0feb10 msgr2=0x7f3bbc0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.875+0000 7f3bc421d700 1 --2- 172.21.15.53:0/915438632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0feb10 0x7f3bbc0fef20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009990 tx=0x7f3bb8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.875+0000 7f3bc421d700 1 -- 172.21.15.53:0/915438632 shutdown_connections 2023-07-14T19:23:43.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.875+0000 7f3bc421d700 1 --2- 172.21.15.53:0/915438632 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bbc100690 0x7f3bbc106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.875+0000 7f3bc421d700 1 --2- 172.21.15.53:0/915438632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0feb10 0x7f3bbc0fef20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.875+0000 7f3bc421d700 1 --2- 172.21.15.53:0/915438632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bbc0ffd00 0x7f3bbc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.875+0000 7f3bc421d700 1 -- 172.21.15.53:0/915438632 >> 172.21.15.53:0/915438632 conn(0x7f3bbc0fa130 msgr2=0x7f3bbc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:43.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.875+0000 7f3bc421d700 1 -- 172.21.15.53:0/915438632 shutdown_connections 2023-07-14T19:23:43.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.875+0000 7f3bc421d700 1 -- 172.21.15.53:0/915438632 wait complete. 2023-07-14T19:23:43.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.876+0000 7f3bc421d700 1 Processor -- start 2023-07-14T19:23:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.876+0000 7f3bc421d700 1 -- start start 2023-07-14T19:23:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.876+0000 7f3bc421d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bbc0feb10 0x7f3bbc1948c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.876+0000 7f3bc421d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bbc0ffd00 0x7f3bbc194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.876+0000 7f3bc421d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc100690 0x7f3bbc19be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.876+0000 7f3bc421d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc19c440 con 0x7f3bbc100690 2023-07-14T19:23:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.876+0000 7f3bc421d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc19c580 con 0x7f3bbc0feb10 2023-07-14T19:23:43.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.876+0000 7f3bc421d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc19c710 con 0x7f3bbc0ffd00 2023-07-14T19:23:43.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc27ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc100690 0x7f3bbc19be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:43.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc27ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc100690 0x7f3bbc19be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50326/0 (socket says 172.21.15.53:50326) 2023-07-14T19:23:43.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc27ba700 1 -- 172.21.15.53:0/2085055905 learned_addr learned my addr 172.21.15.53:0/2085055905 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:43.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc1fb9700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bbc0feb10 0x7f3bbc1948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:43.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc17b8700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bbc0ffd00 0x7f3bbc194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:43.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc27ba700 1 -- 172.21.15.53:0/2085055905 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bbc0ffd00 msgr2=0x7f3bbc194e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc27ba700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bbc0ffd00 0x7f3bbc194e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc27ba700 1 -- 172.21.15.53:0/2085055905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bbc0feb10 msgr2=0x7f3bbc1948c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:43.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc27ba700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bbc0feb10 0x7f3bbc1948c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:43.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc27ba700 1 -- 172.21.15.53:0/2085055905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb80095f0 con 0x7f3bbc100690 2023-07-14T19:23:43.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc27ba700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc100690 0x7f3bbc19be70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4007ee0 tx=0x7f3bb400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:43.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bb2ffd700 1 -- 172.21.15.53:0/2085055905 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4017070 con 0x7f3bbc100690 2023-07-14T19:23:43.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bb2ffd700 1 -- 172.21.15.53:0/2085055905 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3bb400f040 con 0x7f3bbc100690 2023-07-14T19:23:43.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc19c9f0 con 0x7f3bbc100690 2023-07-14T19:23:43.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bb2ffd700 1 -- 172.21.15.53:0/2085055905 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4005580 con 0x7f3bbc100690 2023-07-14T19:23:43.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.877+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc19cf50 con 0x7f3bbc100690 2023-07-14T19:23:43.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.878+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc105a90 con 0x7f3bbc100690 2023-07-14T19:23:43.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.879+0000 7f3bb2ffd700 1 -- 172.21.15.53:0/2085055905 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3bb400db20 con 0x7f3bbc100690 2023-07-14T19:23:43.884 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.880+0000 7f3bb2ffd700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3ba80a11d0 0x7f3ba80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:43.884 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.880+0000 7f3bc1fb9700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3ba80a11d0 0x7f3ba80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:43.884 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.880+0000 7f3bb2ffd700 1 -- 172.21.15.53:0/2085055905 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3bb40cc530 con 0x7f3bbc100690 2023-07-14T19:23:43.884 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.880+0000 7f3bc1fb9700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3ba80a11d0 0x7f3ba80a3680 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800b5c0 tx=0x7f3bb8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:43.884 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:43.883+0000 7f3bb2ffd700 1 -- 172.21.15.53:0/2085055905 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3bb4095d30 con 0x7f3bbc100690 2023-07-14T19:23:44.071 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:23:43.693+0000 7f4d96359700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:23:44.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:23:43.693+0000 7f4d96359700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:23:44.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:23:43.693+0000 7f4d96359700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:23:44.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:23:43.693+0000 7f4d96359700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:23:44.072 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:23:43 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:23:43.693+0000 7f4d96359700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:23:44.119 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.118+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f3bbc060a80 con 0x7f3ba80a11d0 2023-07-14T19:23:44.124 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.122+0000 7f3bb2ffd700 1 -- 172.21.15.53:0/2085055905 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f3bbc060a80 con 0x7f3ba80a11d0 2023-07-14T19:23:44.124 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv_0/710482c1-4c1a-4014-9f1f-684472d59609 2023-07-14T19:23:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3ba80a11d0 msgr2=0x7f3ba80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3ba80a11d0 0x7f3ba80a3680 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800b5c0 tx=0x7f3bb8005fd0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc100690 msgr2=0x7f3bbc19be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc100690 0x7f3bbc19be70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4007ee0 tx=0x7f3bb400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 shutdown_connections 2023-07-14T19:23:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3ba80a11d0 0x7f3ba80a3680 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc100690 0x7f3bbc19be70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3bbc0feb10 0x7f3bbc1948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 --2- 172.21.15.53:0/2085055905 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3bbc0ffd00 0x7f3bbc194e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 >> 172.21.15.53:0/2085055905 conn(0x7f3bbc0fa130 msgr2=0x7f3bbc0fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:44.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 shutdown_connections 2023-07-14T19:23:44.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.125+0000 7f3bc421d700 1 -- 172.21.15.53:0/2085055905 wait complete. 2023-07-14T19:23:44.138 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-07-14T19:23:44.268 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.266+0000 7fbea1270700 1 -- 172.21.15.53:0/382209337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0ffd20 msgr2=0x7fbe9c100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.268 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.266+0000 7fbea1270700 1 --2- 172.21.15.53:0/382209337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0ffd20 0x7fbe9c100170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009990 tx=0x7fbe84009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.268 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.266+0000 7fbea1270700 1 -- 172.21.15.53:0/382209337 shutdown_connections 2023-07-14T19:23:44.268 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.266+0000 7fbea1270700 1 --2- 172.21.15.53:0/382209337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe9c1006b0 0x7fbe9c106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.269 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.266+0000 7fbea1270700 1 --2- 172.21.15.53:0/382209337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe9c0feb30 0x7fbe9c0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.269 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.266+0000 7fbea1270700 1 --2- 172.21.15.53:0/382209337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0ffd20 0x7fbe9c100170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.269 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.266+0000 7fbea1270700 1 -- 172.21.15.53:0/382209337 >> 172.21.15.53:0/382209337 conn(0x7fbe9c0fa190 msgr2=0x7fbe9c0fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:44.269 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.266+0000 7fbea1270700 1 -- 172.21.15.53:0/382209337 shutdown_connections 2023-07-14T19:23:44.269 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.266+0000 7fbea1270700 1 -- 172.21.15.53:0/382209337 wait complete. 2023-07-14T19:23:44.269 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbea1270700 1 Processor -- start 2023-07-14T19:23:44.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbea1270700 1 -- start start 2023-07-14T19:23:44.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbea1270700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0feb30 0x7fbe9c1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbea1270700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe9c0ffd20 0x7fbe9c195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbea1270700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe9c1006b0 0x7fbe9c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbea1270700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c19d520 con 0x7fbe9c0feb30 2023-07-14T19:23:44.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbea1270700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c19d660 con 0x7fbe9c1006b0 2023-07-14T19:23:44.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbea1270700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c19d7f0 con 0x7fbe9c0ffd20 2023-07-14T19:23:44.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbe9affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0feb30 0x7fbe9c1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbe9affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0feb30 0x7fbe9c1959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50336/0 (socket says 172.21.15.53:50336) 2023-07-14T19:23:44.272 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbe9affd700 1 -- 172.21.15.53:0/922101863 learned_addr learned my addr 172.21.15.53:0/922101863 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:44.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.267+0000 7fbe9a7fc700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe9c0ffd20 0x7fbe9c195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe9b7fe700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe9c1006b0 0x7fbe9c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe9affd700 1 -- 172.21.15.53:0/922101863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe9c0ffd20 msgr2=0x7fbe9c195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe9affd700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe9c0ffd20 0x7fbe9c195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe9affd700 1 -- 172.21.15.53:0/922101863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe9c1006b0 msgr2=0x7fbe9c19cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe9affd700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe9c1006b0 0x7fbe9c19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe9affd700 1 -- 172.21.15.53:0/922101863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe840095f0 con 0x7fbe9c0feb30 2023-07-14T19:23:44.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe9affd700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0feb30 0x7fbe9c1959a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c970 tx=0x7fbe8c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:44.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe93fff700 1 -- 172.21.15.53:0/922101863 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c004220 con 0x7fbe9c0feb30 2023-07-14T19:23:44.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c19dad0 con 0x7fbe9c0feb30 2023-07-14T19:23:44.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe93fff700 1 -- 172.21.15.53:0/922101863 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe8c007d60 con 0x7fbe9c0feb30 2023-07-14T19:23:44.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbe93fff700 1 -- 172.21.15.53:0/922101863 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c003c90 con 0x7fbe9c0feb30 2023-07-14T19:23:44.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.268+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c19dfe0 con 0x7fbe9c0feb30 2023-07-14T19:23:44.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.269+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c10a350 con 0x7fbe9c0feb30 2023-07-14T19:23:44.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.270+0000 7fbe93fff700 1 -- 172.21.15.53:0/922101863 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbe8c004380 con 0x7fbe9c0feb30 2023-07-14T19:23:44.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.271+0000 7fbe93fff700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbe880a1150 0x7fbe880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.271+0000 7fbe9a7fc700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbe880a1150 0x7fbe880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.271+0000 7fbe93fff700 1 -- 172.21.15.53:0/922101863 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbe8c0cc330 con 0x7fbe9c0feb30 2023-07-14T19:23:44.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.271+0000 7fbe9a7fc700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbe880a1150 0x7fbe880a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400b5c0 tx=0x7fbe84005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:44.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.275+0000 7fbe93fff700 1 -- 172.21.15.53:0/922101863 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbe8c095bd0 con 0x7fbe9c0feb30 2023-07-14T19:23:44.509 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.507+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe9c069100 con 0x7fbe880a1150 2023-07-14T19:23:44.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.531+0000 7fbe93fff700 1 -- 172.21.15.53:0/922101863 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe9c069100 con 0x7fbe880a1150 2023-07-14T19:23:44.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbe880a1150 msgr2=0x7fbe880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbe880a1150 0x7fbe880a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400b5c0 tx=0x7fbe84005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0feb30 msgr2=0x7fbe9c1959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0feb30 0x7fbe9c1959a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c970 tx=0x7fbe8c00cc80 comp rx=0 tx=0).stop 2023-07-14T19:23:44.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 shutdown_connections 2023-07-14T19:23:44.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbe880a1150 0x7fbe880a3600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe9c1006b0 0x7fbe9c19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9c0feb30 0x7fbe9c1959a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 --2- 172.21.15.53:0/922101863 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe9c0ffd20 0x7fbe9c195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 >> 172.21.15.53:0/922101863 conn(0x7fbe9c0fa190 msgr2=0x7fbe9c0689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:44.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 shutdown_connections 2023-07-14T19:23:44.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.533+0000 7fbea1270700 1 -- 172.21.15.53:0/922101863 wait complete. 2023-07-14T19:23:44.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-07-14T19:23:44.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.673+0000 7f977f382700 1 -- 172.21.15.53:0/272085590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 msgr2=0x7f9778105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.673+0000 7f977f382700 1 --2- 172.21.15.53:0/272085590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 0x7f9778105390 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9774009990 tx=0x7f9774009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.673+0000 7f977f382700 1 -- 172.21.15.53:0/272085590 shutdown_connections 2023-07-14T19:23:44.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.673+0000 7f977f382700 1 --2- 172.21.15.53:0/272085590 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97781058d0 0x7f9778107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.673+0000 7f977f382700 1 --2- 172.21.15.53:0/272085590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9778100690 0x7f9778102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.673+0000 7f977f382700 1 --2- 172.21.15.53:0/272085590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 0x7f9778105390 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.673+0000 7f977f382700 1 -- 172.21.15.53:0/272085590 >> 172.21.15.53:0/272085590 conn(0x7f97780fa130 msgr2=0x7f97780fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:44.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.673+0000 7f977f382700 1 -- 172.21.15.53:0/272085590 shutdown_connections 2023-07-14T19:23:44.679 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.673+0000 7f977f382700 1 -- 172.21.15.53:0/272085590 wait complete. 2023-07-14T19:23:44.679 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977f382700 1 Processor -- start 2023-07-14T19:23:44.679 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977f382700 1 -- start start 2023-07-14T19:23:44.679 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977f382700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778100690 0x7f9778100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.679 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977f382700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 0x7f97780fe640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.679 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977f382700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97781058d0 0x7f97780feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977f382700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97780ff320 con 0x7f9778102fb0 2023-07-14T19:23:44.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977f382700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97780ff490 con 0x7f97781058d0 2023-07-14T19:23:44.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977f382700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977819f990 con 0x7f9778100690 2023-07-14T19:23:44.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977c91d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 0x7f97780fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977c91d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 0x7f97780fe640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50354/0 (socket says 172.21.15.53:50354) 2023-07-14T19:23:44.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977c91d700 1 -- 172.21.15.53:0/2830600786 learned_addr learned my addr 172.21.15.53:0/2830600786 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:44.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977d11e700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778100690 0x7f9778100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.674+0000 7f977d91f700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97781058d0 0x7f97780feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977c91d700 1 -- 172.21.15.53:0/2830600786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778100690 msgr2=0x7f9778100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977c91d700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778100690 0x7f9778100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977c91d700 1 -- 172.21.15.53:0/2830600786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97781058d0 msgr2=0x7f97780feb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977c91d700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97781058d0 0x7f97780feb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977c91d700 1 -- 172.21.15.53:0/2830600786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97740095f0 con 0x7f9778102fb0 2023-07-14T19:23:44.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977d11e700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778100690 0x7f9778100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:23:44.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977d91f700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97781058d0 0x7f97780feb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:23:44.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977c91d700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 0x7f97780fe640 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9774000c00 tx=0x7f977400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:44.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f976e7fc700 1 -- 172.21.15.53:0/2830600786 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977401d070 con 0x7f9778102fb0 2023-07-14T19:23:44.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977819fc10 con 0x7f9778102fb0 2023-07-14T19:23:44.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f976e7fc700 1 -- 172.21.15.53:0/2830600786 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f977400fc00 con 0x7f9778102fb0 2023-07-14T19:23:44.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f976e7fc700 1 -- 172.21.15.53:0/2830600786 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774017650 con 0x7f9778102fb0 2023-07-14T19:23:44.683 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.675+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97781a00c0 con 0x7f9778102fb0 2023-07-14T19:23:44.683 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.676+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977810b3f0 con 0x7f9778102fb0 2023-07-14T19:23:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.677+0000 7f976e7fc700 1 -- 172.21.15.53:0/2830600786 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9774022470 con 0x7f9778102fb0 2023-07-14T19:23:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.677+0000 7f976e7fc700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97640a1150 0x7f97640a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.678+0000 7f977d11e700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97640a1150 0x7f97640a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.678+0000 7f976e7fc700 1 -- 172.21.15.53:0/2830600786 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f97740cddd0 con 0x7f9778102fb0 2023-07-14T19:23:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.678+0000 7f977d11e700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97640a1150 0x7f97640a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9768005950 tx=0x7f97680058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.681+0000 7f976e7fc700 1 -- 172.21.15.53:0/2830600786 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9774097660 con 0x7f9778102fb0 2023-07-14T19:23:44.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.918+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9778102230 con 0x7f97640a1150 2023-07-14T19:23:44.924 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.922+0000 7f976e7fc700 1 -- 172.21.15.53:0/2830600786 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f9778102230 con 0x7f97640a1150 2023-07-14T19:23:44.924 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv_1/92cbb091-800c-4964-9131-34fa4e818ed4 2023-07-14T19:23:44.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97640a1150 msgr2=0x7f97640a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97640a1150 0x7f97640a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9768005950 tx=0x7f97680058e0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 msgr2=0x7f97780fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 0x7f97780fe640 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9774000c00 tx=0x7f977400f710 comp rx=0 tx=0).stop 2023-07-14T19:23:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 shutdown_connections 2023-07-14T19:23:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97640a1150 0x7f97640a3600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97781058d0 0x7f97780feb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9778100690 0x7f9778100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 --2- 172.21.15.53:0/2830600786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9778102fb0 0x7f97780fe640 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 >> 172.21.15.53:0/2830600786 conn(0x7f97780fa130 msgr2=0x7f9778101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:44.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 shutdown_connections 2023-07-14T19:23:44.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:44.924+0000 7f977f382700 1 -- 172.21.15.53:0/2830600786 wait complete. 2023-07-14T19:23:44.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:23:45.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.061+0000 7f150cfd0700 1 -- 172.21.15.53:0/1014521165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15080feb00 msgr2=0x7f15080fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.061+0000 7f150cfd0700 1 --2- 172.21.15.53:0/1014521165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15080feb00 0x7f15080fef10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009990 tx=0x7f14f8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.061+0000 7f150cfd0700 1 -- 172.21.15.53:0/1014521165 shutdown_connections 2023-07-14T19:23:45.065 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.061+0000 7f150cfd0700 1 --2- 172.21.15.53:0/1014521165 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1508100680 0x7f1508106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.065 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.061+0000 7f150cfd0700 1 --2- 172.21.15.53:0/1014521165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15080feb00 0x7f15080fef10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.065 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.061+0000 7f150cfd0700 1 --2- 172.21.15.53:0/1014521165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080ffcf0 0x7f1508100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.065 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.061+0000 7f150cfd0700 1 -- 172.21.15.53:0/1014521165 >> 172.21.15.53:0/1014521165 conn(0x7f15080fa100 msgr2=0x7f15080fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:45.065 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.062+0000 7f150cfd0700 1 -- 172.21.15.53:0/1014521165 shutdown_connections 2023-07-14T19:23:45.065 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.062+0000 7f150cfd0700 1 -- 172.21.15.53:0/1014521165 wait complete. 2023-07-14T19:23:45.067 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.062+0000 7f150cfd0700 1 Processor -- start 2023-07-14T19:23:45.067 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.062+0000 7f150cfd0700 1 -- start start 2023-07-14T19:23:45.067 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.062+0000 7f150cfd0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080feb00 0x7f1508195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.067 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.062+0000 7f150cfd0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15080ffcf0 0x7f1508195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.067 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f150cfd0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1508100680 0x7f150819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f150cfd0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150819d4c0 con 0x7f1508100680 2023-07-14T19:23:45.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f150cfd0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150819d600 con 0x7f15080feb00 2023-07-14T19:23:45.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f150cfd0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150819d790 con 0x7f15080ffcf0 2023-07-14T19:23:45.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1506d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1508100680 0x7f150819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1506d9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1508100680 0x7f150819cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50370/0 (socket says 172.21.15.53:50370) 2023-07-14T19:23:45.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1505d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15080ffcf0 0x7f1508195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1506d9d700 1 -- 172.21.15.53:0/3192641406 learned_addr learned my addr 172.21.15.53:0/3192641406 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:45.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f150659c700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080feb00 0x7f1508195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1506d9d700 1 -- 172.21.15.53:0/3192641406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15080ffcf0 msgr2=0x7f1508195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1506d9d700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15080ffcf0 0x7f1508195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1506d9d700 1 -- 172.21.15.53:0/3192641406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080feb00 msgr2=0x7f1508195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1506d9d700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080feb00 0x7f1508195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1506d9d700 1 -- 172.21.15.53:0/3192641406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f80095f0 con 0x7f1508100680 2023-07-14T19:23:45.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f1506d9d700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1508100680 0x7f150819cef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f150000ef30 tx=0x7f150000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:45.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f14f77fe700 1 -- 172.21.15.53:0/3192641406 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500018070 con 0x7f1508100680 2023-07-14T19:23:45.070 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f14f77fe700 1 -- 172.21.15.53:0/3192641406 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f150000f040 con 0x7f1508100680 2023-07-14T19:23:45.070 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150819da10 con 0x7f1508100680 2023-07-14T19:23:45.070 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f14f77fe700 1 -- 172.21.15.53:0/3192641406 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150000cd00 con 0x7f1508100680 2023-07-14T19:23:45.070 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.063+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150819df20 con 0x7f1508100680 2023-07-14T19:23:45.070 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.064+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1508106bd0 con 0x7f1508100680 2023-07-14T19:23:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.065+0000 7f14f77fe700 1 -- 172.21.15.53:0/3192641406 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1500003a00 con 0x7f1508100680 2023-07-14T19:23:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.066+0000 7f14f77fe700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f14f00a11a0 0x7f14f00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.066+0000 7f14f77fe700 1 -- 172.21.15.53:0/3192641406 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f15000ccc50 con 0x7f1508100680 2023-07-14T19:23:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.066+0000 7f150659c700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f14f00a11a0 0x7f14f00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.072 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.066+0000 7f150659c700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f14f00a11a0 0x7f14f00a3650 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f14f8006010 tx=0x7f14f8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:45.072 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.069+0000 7f14f77fe700 1 -- 172.21.15.53:0/3192641406 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1500096580 con 0x7f1508100680 2023-07-14T19:23:45.293 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:45 smithi053 ceph-mon[96565]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:45.294 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:45 smithi053 ceph-mon[96565]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:45.319 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.317+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f150802caf0 con 0x7f1508100680 2023-07-14T19:23:45.320 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.317+0000 7f14f77fe700 1 -- 172.21.15.53:0/3192641406 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f1500099ba0 con 0x7f1508100680 2023-07-14T19:23:45.320 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:45.320 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:23:37.894565+0000","modified":"2023-07-14T19:23:43.289341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24491,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24484":{"gid":24484,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24491":{"gid":24491,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6839/2496341573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":2496341573},{"type":"v1","addr":"172.21.15.70:6839","nonce":2496341573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:23:45.322 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f14f00a11a0 msgr2=0x7f14f00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.323 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f14f00a11a0 0x7f14f00a3650 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f14f8006010 tx=0x7f14f8005960 comp rx=0 tx=0).stop 2023-07-14T19:23:45.323 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1508100680 msgr2=0x7f150819cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.323 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1508100680 0x7f150819cef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f150000ef30 tx=0x7f150000c3a0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.323 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 shutdown_connections 2023-07-14T19:23:45.324 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f14f00a11a0 0x7f14f00a3650 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.324 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1508100680 0x7f150819cef0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.324 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15080feb00 0x7f1508195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.324 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 --2- 172.21.15.53:0/3192641406 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15080ffcf0 0x7f1508195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.324 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 >> 172.21.15.53:0/3192641406 conn(0x7f15080fa100 msgr2=0x7f15080fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:45.324 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 shutdown_connections 2023-07-14T19:23:45.324 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.320+0000 7f150cfd0700 1 -- 172.21.15.53:0/3192641406 wait complete. 2023-07-14T19:23:45.325 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 23 2023-07-14T19:23:45.335 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:23:45.464 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.462+0000 7f7dba585700 1 -- 172.21.15.53:0/2926262571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 msgr2=0x7f7db41053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.462+0000 7f7dba585700 1 --2- 172.21.15.53:0/2926262571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 0x7f7db41053e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7da8009990 tx=0x7f7da8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.462+0000 7f7dba585700 1 -- 172.21.15.53:0/2926262571 shutdown_connections 2023-07-14T19:23:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.462+0000 7f7dba585700 1 --2- 172.21.15.53:0/2926262571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7db4105920 0x7f7db4107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.462+0000 7f7dba585700 1 --2- 172.21.15.53:0/2926262571 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7db41006e0 0x7f7db4102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.462+0000 7f7dba585700 1 --2- 172.21.15.53:0/2926262571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 0x7f7db41053e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.462+0000 7f7dba585700 1 -- 172.21.15.53:0/2926262571 >> 172.21.15.53:0/2926262571 conn(0x7f7db40fa100 msgr2=0x7f7db40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:45.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.462+0000 7f7dba585700 1 -- 172.21.15.53:0/2926262571 shutdown_connections 2023-07-14T19:23:45.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.462+0000 7f7dba585700 1 -- 172.21.15.53:0/2926262571 wait complete. 2023-07-14T19:23:45.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7dba585700 1 Processor -- start 2023-07-14T19:23:45.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7dba585700 1 -- start start 2023-07-14T19:23:45.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7dba585700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7db41006e0 0x7f7db4100050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7dba585700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 0x7f7db40fe690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7dba585700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7db4105920 0x7f7db40febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7dba585700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db40ff370 con 0x7f7db4103000 2023-07-14T19:23:45.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7dba585700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db40ff4e0 con 0x7f7db4105920 2023-07-14T19:23:45.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7dba585700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db419f970 con 0x7f7db41006e0 2023-07-14T19:23:45.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7db37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 0x7f7db40fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7db37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 0x7f7db40fe690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50382/0 (socket says 172.21.15.53:50382) 2023-07-14T19:23:45.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7db37fe700 1 -- 172.21.15.53:0/1070486337 learned_addr learned my addr 172.21.15.53:0/1070486337 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:45.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7db3fff700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7db41006e0 0x7f7db4100050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7db8b22700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7db4105920 0x7f7db40febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7db37fe700 1 -- 172.21.15.53:0/1070486337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7db41006e0 msgr2=0x7f7db4100050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.463+0000 7f7db37fe700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7db41006e0 0x7f7db4100050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7db37fe700 1 -- 172.21.15.53:0/1070486337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7db4105920 msgr2=0x7f7db40febd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7db37fe700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7db4105920 0x7f7db40febd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7db37fe700 1 -- 172.21.15.53:0/1070486337 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da80095f0 con 0x7f7db4103000 2023-07-14T19:23:45.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7db3fff700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7db41006e0 0x7f7db4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:23:45.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7db37fe700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 0x7f7db40fe690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7da8006010 tx=0x7f7da8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:45.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7db17fa700 1 -- 172.21.15.53:0/1070486337 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da801d070 con 0x7f7db4103000 2023-07-14T19:23:45.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7db17fa700 1 -- 172.21.15.53:0/1070486337 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7da8005030 con 0x7f7db4103000 2023-07-14T19:23:45.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db419fbf0 con 0x7f7db4103000 2023-07-14T19:23:45.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7db17fa700 1 -- 172.21.15.53:0/1070486337 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da8004b90 con 0x7f7db4103000 2023-07-14T19:23:45.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.464+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db419ffb0 con 0x7f7db4103000 2023-07-14T19:23:45.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.465+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db40fbc70 con 0x7f7db4103000 2023-07-14T19:23:45.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.466+0000 7f7db17fa700 1 -- 172.21.15.53:0/1070486337 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7da8004d10 con 0x7f7db4103000 2023-07-14T19:23:45.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.467+0000 7f7db17fa700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7d9c0a12f0 0x7f7d9c0a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.467+0000 7f7db3fff700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7d9c0a12f0 0x7f7d9c0a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.467+0000 7f7db17fa700 1 -- 172.21.15.53:0/1070486337 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7da80cdba0 con 0x7f7db4103000 2023-07-14T19:23:45.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.467+0000 7f7db3fff700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7d9c0a12f0 0x7f7d9c0a37a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7da4011490 tx=0x7f7da4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:45.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.471+0000 7f7db17fa700 1 -- 172.21.15.53:0/1070486337 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7da805a780 con 0x7f7db4103000 2023-07-14T19:23:45.676 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.674+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7db402caf0 con 0x7f7db4103000 2023-07-14T19:23:45.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.675+0000 7f7db17fa700 1 -- 172.21.15.53:0/1070486337 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22054 (secure 0 0 0) 0x7f7da80972d0 con 0x7f7db4103000 2023-07-14T19:23:45.678 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:45.678 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":80,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:23:39.280266+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:23:45.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7d9c0a12f0 msgr2=0x7f7d9c0a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7d9c0a12f0 0x7f7d9c0a37a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7da4011490 tx=0x7f7da4009260 comp rx=0 tx=0).stop 2023-07-14T19:23:45.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 msgr2=0x7f7db40fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 0x7f7db40fe690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7da8006010 tx=0x7f7da8004730 comp rx=0 tx=0).stop 2023-07-14T19:23:45.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 shutdown_connections 2023-07-14T19:23:45.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7d9c0a12f0 0x7f7d9c0a37a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7db4105920 0x7f7db40febd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7db41006e0 0x7f7db4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 --2- 172.21.15.53:0/1070486337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7db4103000 0x7f7db40fe690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.677+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 >> 172.21.15.53:0/1070486337 conn(0x7f7db40fa100 msgr2=0x7f7db4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:45.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.678+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 shutdown_connections 2023-07-14T19:23:45.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.678+0000 7f7dba585700 1 -- 172.21.15.53:0/1070486337 wait complete. 2023-07-14T19:23:45.692 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-14T19:23:45.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T19:23:45.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:45 smithi070 ceph-mon[98015]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:45.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:45 smithi070 ceph-mon[98015]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:45.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:45 smithi104 ceph-mon[98435]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:45.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:45 smithi104 ceph-mon[98435]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:45.818 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.816+0000 7fbec8257700 1 -- 172.21.15.53:0/1597847073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec0100100 msgr2=0x7fbec0100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.819 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.816+0000 7fbec8257700 1 --2- 172.21.15.53:0/1597847073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec0100100 0x7fbec0100510 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009990 tx=0x7fbeb0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.819 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.816+0000 7fbec8257700 1 -- 172.21.15.53:0/1597847073 shutdown_connections 2023-07-14T19:23:45.819 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.816+0000 7fbec8257700 1 --2- 172.21.15.53:0/1597847073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbec00ff700 0x7fbec0106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.819 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.816+0000 7fbec8257700 1 --2- 172.21.15.53:0/1597847073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec0100100 0x7fbec0100510 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.819 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.816+0000 7fbec8257700 1 --2- 172.21.15.53:0/1597847073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbec00fed50 0x7fbec00ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.819 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.816+0000 7fbec8257700 1 -- 172.21.15.53:0/1597847073 >> 172.21.15.53:0/1597847073 conn(0x7fbec00f98f0 msgr2=0x7fbec00fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:45.820 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.816+0000 7fbec8257700 1 -- 172.21.15.53:0/1597847073 shutdown_connections 2023-07-14T19:23:45.820 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.816+0000 7fbec8257700 1 -- 172.21.15.53:0/1597847073 wait complete. 2023-07-14T19:23:45.821 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec8257700 1 Processor -- start 2023-07-14T19:23:45.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec8257700 1 -- start start 2023-07-14T19:23:45.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec8257700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbec00fed50 0x7fbec00fe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec8257700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec00ff700 0x7fbec019a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec8257700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbec0100100 0x7fbec019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec8257700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec019d470 con 0x7fbec00ff700 2023-07-14T19:23:45.823 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec8257700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec019d5e0 con 0x7fbec0100100 2023-07-14T19:23:45.823 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec8257700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec019d7a0 con 0x7fbec00fed50 2023-07-14T19:23:45.823 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec57f2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec00ff700 0x7fbec019a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.823 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec57f2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec00ff700 0x7fbec019a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50394/0 (socket says 172.21.15.53:50394) 2023-07-14T19:23:45.824 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec5ff3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbec00fed50 0x7fbec00fe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.824 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec57f2700 1 -- 172.21.15.53:0/2412308163 learned_addr learned my addr 172.21.15.53:0/2412308163 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:45.824 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec67f4700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbec0100100 0x7fbec019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.824 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec57f2700 1 -- 172.21.15.53:0/2412308163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbec00fed50 msgr2=0x7fbec00fe7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.824 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec57f2700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbec00fed50 0x7fbec00fe7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.824 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec57f2700 1 -- 172.21.15.53:0/2412308163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbec0100100 msgr2=0x7fbec019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:45.825 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec57f2700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbec0100100 0x7fbec019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:45.825 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.817+0000 7fbec57f2700 1 -- 172.21.15.53:0/2412308163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb00095f0 con 0x7fbec00ff700 2023-07-14T19:23:45.825 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.818+0000 7fbec57f2700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec00ff700 0x7fbec019a650 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00d950 tx=0x7fbebc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:45.825 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.818+0000 7fbeb6ffd700 1 -- 172.21.15.53:0/2412308163 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc00f890 con 0x7fbec00ff700 2023-07-14T19:23:45.826 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.818+0000 7fbeb6ffd700 1 -- 172.21.15.53:0/2412308163 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbebc010460 con 0x7fbec00ff700 2023-07-14T19:23:45.826 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.818+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec019da80 con 0x7fbec00ff700 2023-07-14T19:23:45.826 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.818+0000 7fbeb6ffd700 1 -- 172.21.15.53:0/2412308163 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc00e610 con 0x7fbec00ff700 2023-07-14T19:23:45.826 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.818+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec019df90 con 0x7fbec00ff700 2023-07-14T19:23:45.827 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.819+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec0106100 con 0x7fbec00ff700 2023-07-14T19:23:45.828 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.820+0000 7fbeb6ffd700 1 -- 172.21.15.53:0/2412308163 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbebc0108d0 con 0x7fbec00ff700 2023-07-14T19:23:45.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.820+0000 7fbeb6ffd700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbeac0a1220 0x7fbeac0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:45.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.820+0000 7fbeb6ffd700 1 -- 172.21.15.53:0/2412308163 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbebc0cc640 con 0x7fbec00ff700 2023-07-14T19:23:45.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.820+0000 7fbec5ff3700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbeac0a1220 0x7fbeac0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:45.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.821+0000 7fbec5ff3700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbeac0a1220 0x7fbeac0a36d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000b2e0 tx=0x7fbeb0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:45.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:45.824+0000 7fbeb6ffd700 1 -- 172.21.15.53:0/2412308163 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbebc095de0 con 0x7fbec00ff700 2023-07-14T19:23:46.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.016+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbec0060a80 con 0x7fbeac0a1220 2023-07-14T19:23:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.029+0000 7fbeb6ffd700 1 -- 172.21.15.53:0/2412308163 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335987 (secure 0 0 0) 0x7fbec0060a80 con 0x7fbeac0a1220 2023-07-14T19:23:46.035 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:46.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbeac0a1220 msgr2=0x7fbeac0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:46.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbeac0a1220 0x7fbeac0a36d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000b2e0 tx=0x7fbeb0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec00ff700 msgr2=0x7fbec019a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:46.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec00ff700 0x7fbec019a650 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00d950 tx=0x7fbebc00dc60 comp rx=0 tx=0).stop 2023-07-14T19:23:46.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 shutdown_connections 2023-07-14T19:23:46.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fbeac0a1220 0x7fbeac0a36d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbec0100100 0x7fbec019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbec00fed50 0x7fbec00fe7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 --2- 172.21.15.53:0/2412308163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbec00ff700 0x7fbec019a650 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.032+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 >> 172.21.15.53:0/2412308163 conn(0x7fbec00f98f0 msgr2=0x7fbec0103240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.033+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 shutdown_connections 2023-07-14T19:23:46.040 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.033+0000 7fbec8257700 1 -- 172.21.15.53:0/2412308163 wait complete. 2023-07-14T19:23:46.040 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2023-07-14T19:23:46.047 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2023-07-14T19:23:45.315763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":629578,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":164,"num_read_kb":158,"num_write":293,"num_write_kb":3003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":277,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336276,"kb_used_data":12056,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605548,"statfs":{"total":1151940427776,"available":1151596081152,"internally_reserved":0,"allocated":12345344,"data_stored":6815016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39210,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":28,"num_read_kb":39,"num_write":112,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.870748"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472803+0000","last_change":"2023-07-14T19:23:38.301785+0000","last_active":"2023-07-14T19:23:39.472803+0000","last_peered":"2023-07-14T19:23:39.472803+0000","last_clean":"2023-07-14T19:23:39.472803+0000","last_became_active":"2023-07-14T19:23:38.301014+0000","last_became_peered":"2023-07-14T19:23:38.301014+0000","last_unstale":"2023-07-14T19:23:39.472803+0000","last_undegraded":"2023-07-14T19:23:39.472803+0000","last_fullsized":"2023-07-14T19:23:39.472803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:02.663606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514628+0000","last_change":"2023-07-14T19:23:38.299101+0000","last_active":"2023-07-14T19:23:39.514628+0000","last_peered":"2023-07-14T19:23:39.514628+0000","last_clean":"2023-07-14T19:23:39.514628+0000","last_became_active":"2023-07-14T19:23:38.298720+0000","last_became_peered":"2023-07-14T19:23:38.298720+0000","last_unstale":"2023-07-14T19:23:39.514628+0000","last_undegraded":"2023-07-14T19:23:39.514628+0000","last_fullsized":"2023-07-14T19:23:39.514628+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:05:31.873298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514167+0000","last_change":"2023-07-14T19:23:38.295335+0000","last_active":"2023-07-14T19:23:39.514167+0000","last_peered":"2023-07-14T19:23:39.514167+0000","last_clean":"2023-07-14T19:23:39.514167+0000","last_became_active":"2023-07-14T19:23:38.295031+0000","last_became_peered":"2023-07-14T19:23:38.295031+0000","last_unstale":"2023-07-14T19:23:39.514167+0000","last_undegraded":"2023-07-14T19:23:39.514167+0000","last_fullsized":"2023-07-14T19:23:39.514167+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:09.443353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515703+0000","last_change":"2023-07-14T19:23:38.295055+0000","last_active":"2023-07-14T19:23:39.515703+0000","last_peered":"2023-07-14T19:23:39.515703+0000","last_clean":"2023-07-14T19:23:39.515703+0000","last_became_active":"2023-07-14T19:23:38.294751+0000","last_became_peered":"2023-07-14T19:23:38.294751+0000","last_unstale":"2023-07-14T19:23:39.515703+0000","last_undegraded":"2023-07-14T19:23:39.515703+0000","last_fullsized":"2023-07-14T19:23:39.515703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:19.020190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286436+0000","last_change":"2023-07-14T19:23:38.294684+0000","last_active":"2023-07-14T19:23:39.286436+0000","last_peered":"2023-07-14T19:23:39.286436+0000","last_clean":"2023-07-14T19:23:39.286436+0000","last_became_active":"2023-07-14T19:23:38.294368+0000","last_became_peered":"2023-07-14T19:23:38.294368+0000","last_unstale":"2023-07-14T19:23:39.286436+0000","last_undegraded":"2023-07-14T19:23:39.286436+0000","last_fullsized":"2023-07-14T19:23:39.286436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:35.017660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283944+0000","last_change":"2023-07-14T19:23:38.301219+0000","last_active":"2023-07-14T19:23:39.283944+0000","last_peered":"2023-07-14T19:23:39.283944+0000","last_clean":"2023-07-14T19:23:39.283944+0000","last_became_active":"2023-07-14T19:23:38.300904+0000","last_became_peered":"2023-07-14T19:23:38.300904+0000","last_unstale":"2023-07-14T19:23:39.283944+0000","last_undegraded":"2023-07-14T19:23:39.283944+0000","last_fullsized":"2023-07-14T19:23:39.283944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:53:02.603728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515586+0000","last_change":"2023-07-14T19:23:38.292831+0000","last_active":"2023-07-14T19:23:39.515586+0000","last_peered":"2023-07-14T19:23:39.515586+0000","last_clean":"2023-07-14T19:23:39.515586+0000","last_became_active":"2023-07-14T19:23:38.292536+0000","last_became_peered":"2023-07-14T19:23:38.292536+0000","last_unstale":"2023-07-14T19:23:39.515586+0000","last_undegraded":"2023-07-14T19:23:39.515586+0000","last_fullsized":"2023-07-14T19:23:39.515586+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:49.316491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283393+0000","last_change":"2023-07-14T19:23:38.304940+0000","last_active":"2023-07-14T19:23:39.283393+0000","last_peered":"2023-07-14T19:23:39.283393+0000","last_clean":"2023-07-14T19:23:39.283393+0000","last_became_active":"2023-07-14T19:23:38.304609+0000","last_became_peered":"2023-07-14T19:23:38.304609+0000","last_unstale":"2023-07-14T19:23:39.283393+0000","last_undegraded":"2023-07-14T19:23:39.283393+0000","last_fullsized":"2023-07-14T19:23:39.283393+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:14.673980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515328+0000","last_change":"2023-07-14T19:23:38.300273+0000","last_active":"2023-07-14T19:23:39.515328+0000","last_peered":"2023-07-14T19:23:39.515328+0000","last_clean":"2023-07-14T19:23:39.515328+0000","last_became_active":"2023-07-14T19:23:38.299965+0000","last_became_peered":"2023-07-14T19:23:38.299965+0000","last_unstale":"2023-07-14T19:23:39.515328+0000","last_undegraded":"2023-07-14T19:23:39.515328+0000","last_fullsized":"2023-07-14T19:23:39.515328+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:26:21.737406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284243+0000","last_change":"2023-07-14T19:23:38.302757+0000","last_active":"2023-07-14T19:23:39.284243+0000","last_peered":"2023-07-14T19:23:39.284243+0000","last_clean":"2023-07-14T19:23:39.284243+0000","last_became_active":"2023-07-14T19:23:38.302409+0000","last_became_peered":"2023-07-14T19:23:38.302409+0000","last_unstale":"2023-07-14T19:23:39.284243+0000","last_undegraded":"2023-07-14T19:23:39.284243+0000","last_fullsized":"2023-07-14T19:23:39.284243+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:09:43.204311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515622+0000","last_change":"2023-07-14T19:23:38.296974+0000","last_active":"2023-07-14T19:23:39.515622+0000","last_peered":"2023-07-14T19:23:39.515622+0000","last_clean":"2023-07-14T19:23:39.515622+0000","last_became_active":"2023-07-14T19:23:38.296718+0000","last_became_peered":"2023-07-14T19:23:38.296718+0000","last_unstale":"2023-07-14T19:23:39.515622+0000","last_undegraded":"2023-07-14T19:23:39.515622+0000","last_fullsized":"2023-07-14T19:23:39.515622+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:19:13.680031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515564+0000","last_change":"2023-07-14T19:23:38.297575+0000","last_active":"2023-07-14T19:23:39.515564+0000","last_peered":"2023-07-14T19:23:39.515564+0000","last_clean":"2023-07-14T19:23:39.515564+0000","last_became_active":"2023-07-14T19:23:38.297292+0000","last_became_peered":"2023-07-14T19:23:38.297292+0000","last_unstale":"2023-07-14T19:23:39.515564+0000","last_undegraded":"2023-07-14T19:23:39.515564+0000","last_fullsized":"2023-07-14T19:23:39.515564+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:49.161960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515373+0000","last_change":"2023-07-14T19:23:38.295810+0000","last_active":"2023-07-14T19:23:39.515373+0000","last_peered":"2023-07-14T19:23:39.515373+0000","last_clean":"2023-07-14T19:23:39.515373+0000","last_became_active":"2023-07-14T19:23:38.295281+0000","last_became_peered":"2023-07-14T19:23:38.295281+0000","last_unstale":"2023-07-14T19:23:39.515373+0000","last_undegraded":"2023-07-14T19:23:39.515373+0000","last_fullsized":"2023-07-14T19:23:39.515373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:37.869421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514485+0000","last_change":"2023-07-14T19:23:38.299271+0000","last_active":"2023-07-14T19:23:39.514485+0000","last_peered":"2023-07-14T19:23:39.514485+0000","last_clean":"2023-07-14T19:23:39.514485+0000","last_became_active":"2023-07-14T19:23:38.298298+0000","last_became_peered":"2023-07-14T19:23:38.298298+0000","last_unstale":"2023-07-14T19:23:39.514485+0000","last_undegraded":"2023-07-14T19:23:39.514485+0000","last_fullsized":"2023-07-14T19:23:39.514485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:30.297961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514219+0000","last_change":"2023-07-14T19:23:38.296295+0000","last_active":"2023-07-14T19:23:39.514219+0000","last_peered":"2023-07-14T19:23:39.514219+0000","last_clean":"2023-07-14T19:23:39.514219+0000","last_became_active":"2023-07-14T19:23:38.296004+0000","last_became_peered":"2023-07-14T19:23:38.296004+0000","last_unstale":"2023-07-14T19:23:39.514219+0000","last_undegraded":"2023-07-14T19:23:39.514219+0000","last_fullsized":"2023-07-14T19:23:39.514219+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:34:50.891851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514438+0000","last_change":"2023-07-14T19:23:38.293292+0000","last_active":"2023-07-14T19:23:39.514438+0000","last_peered":"2023-07-14T19:23:39.514438+0000","last_clean":"2023-07-14T19:23:39.514438+0000","last_became_active":"2023-07-14T19:23:38.292980+0000","last_became_peered":"2023-07-14T19:23:38.292980+0000","last_unstale":"2023-07-14T19:23:39.514438+0000","last_undegraded":"2023-07-14T19:23:39.514438+0000","last_fullsized":"2023-07-14T19:23:39.514438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:22.060162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472781+0000","last_change":"2023-07-14T19:23:38.301950+0000","last_active":"2023-07-14T19:23:39.472781+0000","last_peered":"2023-07-14T19:23:39.472781+0000","last_clean":"2023-07-14T19:23:39.472781+0000","last_became_active":"2023-07-14T19:23:38.301416+0000","last_became_peered":"2023-07-14T19:23:38.301416+0000","last_unstale":"2023-07-14T19:23:39.472781+0000","last_undegraded":"2023-07-14T19:23:39.472781+0000","last_fullsized":"2023-07-14T19:23:39.472781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:03:11.369261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515639+0000","last_change":"2023-07-14T19:23:38.300108+0000","last_active":"2023-07-14T19:23:39.515639+0000","last_peered":"2023-07-14T19:23:39.515639+0000","last_clean":"2023-07-14T19:23:39.515639+0000","last_became_active":"2023-07-14T19:23:38.299773+0000","last_became_peered":"2023-07-14T19:23:38.299773+0000","last_unstale":"2023-07-14T19:23:39.515639+0000","last_undegraded":"2023-07-14T19:23:39.515639+0000","last_fullsized":"2023-07-14T19:23:39.515639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:59.607746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515238+0000","last_change":"2023-07-14T19:23:38.299124+0000","last_active":"2023-07-14T19:23:39.515238+0000","last_peered":"2023-07-14T19:23:39.515238+0000","last_clean":"2023-07-14T19:23:39.515238+0000","last_became_active":"2023-07-14T19:23:38.298744+0000","last_became_peered":"2023-07-14T19:23:38.298744+0000","last_unstale":"2023-07-14T19:23:39.515238+0000","last_undegraded":"2023-07-14T19:23:39.515238+0000","last_fullsized":"2023-07-14T19:23:39.515238+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:09.383666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515197+0000","last_change":"2023-07-14T19:23:37.466299+0000","last_active":"2023-07-14T19:23:39.515197+0000","last_peered":"2023-07-14T19:23:39.515197+0000","last_clean":"2023-07-14T19:23:39.515197+0000","last_became_active":"2023-07-14T19:23:37.465989+0000","last_became_peered":"2023-07-14T19:23:37.465989+0000","last_unstale":"2023-07-14T19:23:39.515197+0000","last_undegraded":"2023-07-14T19:23:39.515197+0000","last_fullsized":"2023-07-14T19:23:39.515197+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:28.346196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283813+0000","last_change":"2023-07-14T19:23:38.301743+0000","last_active":"2023-07-14T19:23:39.283813+0000","last_peered":"2023-07-14T19:23:39.283813+0000","last_clean":"2023-07-14T19:23:39.283813+0000","last_became_active":"2023-07-14T19:23:38.301419+0000","last_became_peered":"2023-07-14T19:23:38.301419+0000","last_unstale":"2023-07-14T19:23:39.283813+0000","last_undegraded":"2023-07-14T19:23:39.283813+0000","last_fullsized":"2023-07-14T19:23:39.283813+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:37.577409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283986+0000","last_change":"2023-07-14T19:23:37.459314+0000","last_active":"2023-07-14T19:23:39.283986+0000","last_peered":"2023-07-14T19:23:39.283986+0000","last_clean":"2023-07-14T19:23:39.283986+0000","last_became_active":"2023-07-14T19:23:37.458925+0000","last_became_peered":"2023-07-14T19:23:37.458925+0000","last_unstale":"2023-07-14T19:23:39.283986+0000","last_undegraded":"2023-07-14T19:23:39.283986+0000","last_fullsized":"2023-07-14T19:23:39.283986+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:59.317422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514675+0000","last_change":"2023-07-14T19:23:38.294432+0000","last_active":"2023-07-14T19:23:39.514675+0000","last_peered":"2023-07-14T19:23:39.514675+0000","last_clean":"2023-07-14T19:23:39.514675+0000","last_became_active":"2023-07-14T19:23:38.294118+0000","last_became_peered":"2023-07-14T19:23:38.294118+0000","last_unstale":"2023-07-14T19:23:39.514675+0000","last_undegraded":"2023-07-14T19:23:39.514675+0000","last_fullsized":"2023-07-14T19:23:39.514675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:48:03.277272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295257+0000","last_change":"2023-07-14T19:23:37.463231+0000","last_active":"2023-07-14T19:23:42.295257+0000","last_peered":"2023-07-14T19:23:42.295257+0000","last_clean":"2023-07-14T19:23:42.295257+0000","last_became_active":"2023-07-14T19:23:37.462571+0000","last_became_peered":"2023-07-14T19:23:37.462571+0000","last_unstale":"2023-07-14T19:23:42.295257+0000","last_undegraded":"2023-07-14T19:23:42.295257+0000","last_fullsized":"2023-07-14T19:23:42.295257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:07:42.060866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515814+0000","last_change":"2023-07-14T19:23:38.300276+0000","last_active":"2023-07-14T19:23:39.515814+0000","last_peered":"2023-07-14T19:23:39.515814+0000","last_clean":"2023-07-14T19:23:39.515814+0000","last_became_active":"2023-07-14T19:23:38.299984+0000","last_became_peered":"2023-07-14T19:23:38.299984+0000","last_unstale":"2023-07-14T19:23:39.515814+0000","last_undegraded":"2023-07-14T19:23:39.515814+0000","last_fullsized":"2023-07-14T19:23:39.515814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:19:10.316989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286669+0000","last_change":"2023-07-14T19:23:37.454592+0000","last_active":"2023-07-14T19:23:39.286669+0000","last_peered":"2023-07-14T19:23:39.286669+0000","last_clean":"2023-07-14T19:23:39.286669+0000","last_became_active":"2023-07-14T19:23:37.454266+0000","last_became_peered":"2023-07-14T19:23:37.454266+0000","last_unstale":"2023-07-14T19:23:39.286669+0000","last_undegraded":"2023-07-14T19:23:39.286669+0000","last_fullsized":"2023-07-14T19:23:39.286669+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:14.362148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.516093+0000","last_change":"2023-07-14T19:23:38.297945+0000","last_active":"2023-07-14T19:23:39.516093+0000","last_peered":"2023-07-14T19:23:39.516093+0000","last_clean":"2023-07-14T19:23:39.516093+0000","last_became_active":"2023-07-14T19:23:38.296733+0000","last_became_peered":"2023-07-14T19:23:38.296733+0000","last_unstale":"2023-07-14T19:23:39.516093+0000","last_undegraded":"2023-07-14T19:23:39.516093+0000","last_fullsized":"2023-07-14T19:23:39.516093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:38:24.778683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.288928+0000","last_change":"2023-07-14T19:23:37.460329+0000","last_active":"2023-07-14T19:23:42.288928+0000","last_peered":"2023-07-14T19:23:42.288928+0000","last_clean":"2023-07-14T19:23:42.288928+0000","last_became_active":"2023-07-14T19:23:37.460037+0000","last_became_peered":"2023-07-14T19:23:37.460037+0000","last_unstale":"2023-07-14T19:23:42.288928+0000","last_undegraded":"2023-07-14T19:23:42.288928+0000","last_fullsized":"2023-07-14T19:23:42.288928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:25:02.768120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286579+0000","last_change":"2023-07-14T19:23:38.293216+0000","last_active":"2023-07-14T19:23:39.286579+0000","last_peered":"2023-07-14T19:23:39.286579+0000","last_clean":"2023-07-14T19:23:39.286579+0000","last_became_active":"2023-07-14T19:23:38.292748+0000","last_became_peered":"2023-07-14T19:23:38.292748+0000","last_unstale":"2023-07-14T19:23:39.286579+0000","last_undegraded":"2023-07-14T19:23:39.286579+0000","last_fullsized":"2023-07-14T19:23:39.286579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:48.829793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295199+0000","last_change":"2023-07-14T19:23:37.462337+0000","last_active":"2023-07-14T19:23:42.295199+0000","last_peered":"2023-07-14T19:23:42.295199+0000","last_clean":"2023-07-14T19:23:42.295199+0000","last_became_active":"2023-07-14T19:23:37.462046+0000","last_became_peered":"2023-07-14T19:23:37.462046+0000","last_unstale":"2023-07-14T19:23:42.295199+0000","last_undegraded":"2023-07-14T19:23:42.295199+0000","last_fullsized":"2023-07-14T19:23:42.295199+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:07:29.186049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.516281+0000","last_change":"2023-07-14T19:23:38.297839+0000","last_active":"2023-07-14T19:23:39.516281+0000","last_peered":"2023-07-14T19:23:39.516281+0000","last_clean":"2023-07-14T19:23:39.516281+0000","last_became_active":"2023-07-14T19:23:38.296915+0000","last_became_peered":"2023-07-14T19:23:38.296915+0000","last_unstale":"2023-07-14T19:23:39.516281+0000","last_undegraded":"2023-07-14T19:23:39.516281+0000","last_fullsized":"2023-07-14T19:23:39.516281+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:44:44.107176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296502+0000","last_change":"2023-07-14T19:23:37.457637+0000","last_active":"2023-07-14T19:23:42.296502+0000","last_peered":"2023-07-14T19:23:42.296502+0000","last_clean":"2023-07-14T19:23:42.296502+0000","last_became_active":"2023-07-14T19:23:37.457305+0000","last_became_peered":"2023-07-14T19:23:37.457305+0000","last_unstale":"2023-07-14T19:23:42.296502+0000","last_undegraded":"2023-07-14T19:23:42.296502+0000","last_fullsized":"2023-07-14T19:23:42.296502+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:08:30.695027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283305+0000","last_change":"2023-07-14T19:23:38.295974+0000","last_active":"2023-07-14T19:23:39.283305+0000","last_peered":"2023-07-14T19:23:39.283305+0000","last_clean":"2023-07-14T19:23:39.283305+0000","last_became_active":"2023-07-14T19:23:38.295641+0000","last_became_peered":"2023-07-14T19:23:38.295641+0000","last_unstale":"2023-07-14T19:23:39.283305+0000","last_undegraded":"2023-07-14T19:23:39.283305+0000","last_fullsized":"2023-07-14T19:23:39.283305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:18:13.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515270+0000","last_change":"2023-07-14T19:23:37.457041+0000","last_active":"2023-07-14T19:23:39.515270+0000","last_peered":"2023-07-14T19:23:39.515270+0000","last_clean":"2023-07-14T19:23:39.515270+0000","last_became_active":"2023-07-14T19:23:37.456454+0000","last_became_peered":"2023-07-14T19:23:37.456454+0000","last_unstale":"2023-07-14T19:23:39.515270+0000","last_undegraded":"2023-07-14T19:23:39.515270+0000","last_fullsized":"2023-07-14T19:23:39.515270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:10:01.796045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515109+0000","last_change":"2023-07-14T19:23:38.293749+0000","last_active":"2023-07-14T19:23:39.515109+0000","last_peered":"2023-07-14T19:23:39.515109+0000","last_clean":"2023-07-14T19:23:39.515109+0000","last_became_active":"2023-07-14T19:23:38.293400+0000","last_became_peered":"2023-07-14T19:23:38.293400+0000","last_unstale":"2023-07-14T19:23:39.515109+0000","last_undegraded":"2023-07-14T19:23:39.515109+0000","last_fullsized":"2023-07-14T19:23:39.515109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:53.267258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'3","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:44.292441+0000","last_change":"2023-07-14T19:23:37.463783+0000","last_active":"2023-07-14T19:23:44.292441+0000","last_peered":"2023-07-14T19:23:44.292441+0000","last_clean":"2023-07-14T19:23:44.292441+0000","last_became_active":"2023-07-14T19:23:37.463421+0000","last_became_peered":"2023-07-14T19:23:37.463421+0000","last_unstale":"2023-07-14T19:23:44.292441+0000","last_undegraded":"2023-07-14T19:23:44.292441+0000","last_fullsized":"2023-07-14T19:23:44.292441+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:38.946539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515132+0000","last_change":"2023-07-14T19:23:38.300956+0000","last_active":"2023-07-14T19:23:39.515132+0000","last_peered":"2023-07-14T19:23:39.515132+0000","last_clean":"2023-07-14T19:23:39.515132+0000","last_became_active":"2023-07-14T19:23:38.300662+0000","last_became_peered":"2023-07-14T19:23:38.300662+0000","last_unstale":"2023-07-14T19:23:39.515132+0000","last_undegraded":"2023-07-14T19:23:39.515132+0000","last_fullsized":"2023-07-14T19:23:39.515132+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:59:21.546322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.476358+0000","last_change":"2023-07-14T19:23:37.458221+0000","last_active":"2023-07-14T19:23:41.476358+0000","last_peered":"2023-07-14T19:23:41.476358+0000","last_clean":"2023-07-14T19:23:41.476358+0000","last_became_active":"2023-07-14T19:23:37.457923+0000","last_became_peered":"2023-07-14T19:23:37.457923+0000","last_unstale":"2023-07-14T19:23:41.476358+0000","last_undegraded":"2023-07-14T19:23:41.476358+0000","last_fullsized":"2023-07-14T19:23:41.476358+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:51.733334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514538+0000","last_change":"2023-07-14T19:23:38.303977+0000","last_active":"2023-07-14T19:23:39.514538+0000","last_peered":"2023-07-14T19:23:39.514538+0000","last_clean":"2023-07-14T19:23:39.514538+0000","last_became_active":"2023-07-14T19:23:38.303333+0000","last_became_peered":"2023-07-14T19:23:38.303333+0000","last_unstale":"2023-07-14T19:23:39.514538+0000","last_undegraded":"2023-07-14T19:23:39.514538+0000","last_fullsized":"2023-07-14T19:23:39.514538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:28.958043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475704+0000","last_change":"2023-07-14T19:23:37.459181+0000","last_active":"2023-07-14T19:23:41.475704+0000","last_peered":"2023-07-14T19:23:41.475704+0000","last_clean":"2023-07-14T19:23:41.475704+0000","last_became_active":"2023-07-14T19:23:37.458890+0000","last_became_peered":"2023-07-14T19:23:37.458890+0000","last_unstale":"2023-07-14T19:23:41.475704+0000","last_undegraded":"2023-07-14T19:23:41.475704+0000","last_fullsized":"2023-07-14T19:23:41.475704+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:54.069559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515862+0000","last_change":"2023-07-14T19:23:38.295057+0000","last_active":"2023-07-14T19:23:39.515862+0000","last_peered":"2023-07-14T19:23:39.515862+0000","last_clean":"2023-07-14T19:23:39.515862+0000","last_became_active":"2023-07-14T19:23:38.294774+0000","last_became_peered":"2023-07-14T19:23:38.294774+0000","last_unstale":"2023-07-14T19:23:39.515862+0000","last_undegraded":"2023-07-14T19:23:39.515862+0000","last_fullsized":"2023-07-14T19:23:39.515862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:50:17.296593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'18","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:45.298945+0000","last_change":"2023-07-14T19:23:37.463196+0000","last_active":"2023-07-14T19:23:45.298945+0000","last_peered":"2023-07-14T19:23:45.298945+0000","last_clean":"2023-07-14T19:23:45.298945+0000","last_became_active":"2023-07-14T19:23:37.462867+0000","last_became_peered":"2023-07-14T19:23:37.462867+0000","last_unstale":"2023-07-14T19:23:45.298945+0000","last_undegraded":"2023-07-14T19:23:45.298945+0000","last_fullsized":"2023-07-14T19:23:45.298945+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:35.156443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36237,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":23,"num_write":19,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286612+0000","last_change":"2023-07-14T19:23:38.293249+0000","last_active":"2023-07-14T19:23:39.286612+0000","last_peered":"2023-07-14T19:23:39.286612+0000","last_clean":"2023-07-14T19:23:39.286612+0000","last_became_active":"2023-07-14T19:23:38.292938+0000","last_became_peered":"2023-07-14T19:23:38.292938+0000","last_unstale":"2023-07-14T19:23:39.286612+0000","last_undegraded":"2023-07-14T19:23:39.286612+0000","last_fullsized":"2023-07-14T19:23:39.286612+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:49:19.220723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"80'6","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296103+0000","last_change":"2023-07-14T19:23:37.460336+0000","last_active":"2023-07-14T19:23:42.296103+0000","last_peered":"2023-07-14T19:23:42.296103+0000","last_clean":"2023-07-14T19:23:42.296103+0000","last_became_active":"2023-07-14T19:23:37.460008+0000","last_became_peered":"2023-07-14T19:23:37.460008+0000","last_unstale":"2023-07-14T19:23:42.296103+0000","last_undegraded":"2023-07-14T19:23:42.296103+0000","last_fullsized":"2023-07-14T19:23:42.296103+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:31.610459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514712+0000","last_change":"2023-07-14T19:23:38.296819+0000","last_active":"2023-07-14T19:23:39.514712+0000","last_peered":"2023-07-14T19:23:39.514712+0000","last_clean":"2023-07-14T19:23:39.514712+0000","last_became_active":"2023-07-14T19:23:38.296225+0000","last_became_peered":"2023-07-14T19:23:38.296225+0000","last_unstale":"2023-07-14T19:23:39.514712+0000","last_undegraded":"2023-07-14T19:23:39.514712+0000","last_fullsized":"2023-07-14T19:23:39.514712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:43:30.925553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474332+0000","last_change":"2023-07-14T19:23:37.460018+0000","last_active":"2023-07-14T19:23:41.474332+0000","last_peered":"2023-07-14T19:23:41.474332+0000","last_clean":"2023-07-14T19:23:41.474332+0000","last_became_active":"2023-07-14T19:23:37.459713+0000","last_became_peered":"2023-07-14T19:23:37.459713+0000","last_unstale":"2023-07-14T19:23:41.474332+0000","last_undegraded":"2023-07-14T19:23:41.474332+0000","last_fullsized":"2023-07-14T19:23:41.474332+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:11.451517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515474+0000","last_change":"2023-07-14T19:23:38.301568+0000","last_active":"2023-07-14T19:23:39.515474+0000","last_peered":"2023-07-14T19:23:39.515474+0000","last_clean":"2023-07-14T19:23:39.515474+0000","last_became_active":"2023-07-14T19:23:38.301279+0000","last_became_peered":"2023-07-14T19:23:38.301279+0000","last_unstale":"2023-07-14T19:23:39.515474+0000","last_undegraded":"2023-07-14T19:23:39.515474+0000","last_fullsized":"2023-07-14T19:23:39.515474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:18.485184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514818+0000","last_change":"2023-07-14T19:23:37.460782+0000","last_active":"2023-07-14T19:23:39.514818+0000","last_peered":"2023-07-14T19:23:39.514818+0000","last_clean":"2023-07-14T19:23:39.514818+0000","last_became_active":"2023-07-14T19:23:37.460457+0000","last_became_peered":"2023-07-14T19:23:37.460457+0000","last_unstale":"2023-07-14T19:23:39.514818+0000","last_undegraded":"2023-07-14T19:23:39.514818+0000","last_fullsized":"2023-07-14T19:23:39.514818+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:19.994091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514572+0000","last_change":"2023-07-14T19:23:38.299681+0000","last_active":"2023-07-14T19:23:39.514572+0000","last_peered":"2023-07-14T19:23:39.514572+0000","last_clean":"2023-07-14T19:23:39.514572+0000","last_became_active":"2023-07-14T19:23:38.299345+0000","last_became_peered":"2023-07-14T19:23:38.299345+0000","last_unstale":"2023-07-14T19:23:39.514572+0000","last_undegraded":"2023-07-14T19:23:39.514572+0000","last_fullsized":"2023-07-14T19:23:39.514572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:58:44.871954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"80'15","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296518+0000","last_change":"2023-07-14T19:23:37.458413+0000","last_active":"2023-07-14T19:23:42.296518+0000","last_peered":"2023-07-14T19:23:42.296518+0000","last_clean":"2023-07-14T19:23:42.296518+0000","last_became_active":"2023-07-14T19:23:37.458108+0000","last_became_peered":"2023-07-14T19:23:37.458108+0000","last_unstale":"2023-07-14T19:23:42.296518+0000","last_undegraded":"2023-07-14T19:23:42.296518+0000","last_fullsized":"2023-07-14T19:23:42.296518+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:36.538174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515638+0000","last_change":"2023-07-14T19:23:38.296350+0000","last_active":"2023-07-14T19:23:39.515638+0000","last_peered":"2023-07-14T19:23:39.515638+0000","last_clean":"2023-07-14T19:23:39.515638+0000","last_became_active":"2023-07-14T19:23:38.296036+0000","last_became_peered":"2023-07-14T19:23:38.296036+0000","last_unstale":"2023-07-14T19:23:39.515638+0000","last_undegraded":"2023-07-14T19:23:39.515638+0000","last_fullsized":"2023-07-14T19:23:39.515638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:58:58.339059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.924053+0000","last_change":"2023-07-14T19:23:37.461647+0000","last_active":"2023-07-14T19:23:39.924053+0000","last_peered":"2023-07-14T19:23:39.924053+0000","last_clean":"2023-07-14T19:23:39.924053+0000","last_became_active":"2023-07-14T19:23:37.461325+0000","last_became_peered":"2023-07-14T19:23:37.461325+0000","last_unstale":"2023-07-14T19:23:39.924053+0000","last_undegraded":"2023-07-14T19:23:39.924053+0000","last_fullsized":"2023-07-14T19:23:39.924053+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:46.235600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472796+0000","last_change":"2023-07-14T19:23:38.300940+0000","last_active":"2023-07-14T19:23:39.472796+0000","last_peered":"2023-07-14T19:23:39.472796+0000","last_clean":"2023-07-14T19:23:39.472796+0000","last_became_active":"2023-07-14T19:23:38.300578+0000","last_became_peered":"2023-07-14T19:23:38.300578+0000","last_unstale":"2023-07-14T19:23:39.472796+0000","last_undegraded":"2023-07-14T19:23:39.472796+0000","last_fullsized":"2023-07-14T19:23:39.472796+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:50.864591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515474+0000","last_change":"2023-07-14T19:23:37.464251+0000","last_active":"2023-07-14T19:23:39.515474+0000","last_peered":"2023-07-14T19:23:39.515474+0000","last_clean":"2023-07-14T19:23:39.515474+0000","last_became_active":"2023-07-14T19:23:37.463793+0000","last_became_peered":"2023-07-14T19:23:37.463793+0000","last_unstale":"2023-07-14T19:23:39.515474+0000","last_undegraded":"2023-07-14T19:23:39.515474+0000","last_fullsized":"2023-07-14T19:23:39.515474+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:28:41.697534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:40.299231+0000","last_change":"2023-07-14T19:23:38.300077+0000","last_active":"2023-07-14T19:23:40.299231+0000","last_peered":"2023-07-14T19:23:40.299231+0000","last_clean":"2023-07-14T19:23:40.299231+0000","last_became_active":"2023-07-14T19:23:38.298708+0000","last_became_peered":"2023-07-14T19:23:38.298708+0000","last_unstale":"2023-07-14T19:23:40.299231+0000","last_undegraded":"2023-07-14T19:23:40.299231+0000","last_fullsized":"2023-07-14T19:23:40.299231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:13:31.564130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515610+0000","last_change":"2023-07-14T19:23:37.458361+0000","last_active":"2023-07-14T19:23:39.515610+0000","last_peered":"2023-07-14T19:23:39.515610+0000","last_clean":"2023-07-14T19:23:39.515610+0000","last_became_active":"2023-07-14T19:23:37.458108+0000","last_became_peered":"2023-07-14T19:23:37.458108+0000","last_unstale":"2023-07-14T19:23:39.515610+0000","last_undegraded":"2023-07-14T19:23:39.515610+0000","last_fullsized":"2023-07-14T19:23:39.515610+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:41:17.858983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284311+0000","last_change":"2023-07-14T19:23:38.301723+0000","last_active":"2023-07-14T19:23:39.284311+0000","last_peered":"2023-07-14T19:23:39.284311+0000","last_clean":"2023-07-14T19:23:39.284311+0000","last_became_active":"2023-07-14T19:23:38.301424+0000","last_became_peered":"2023-07-14T19:23:38.301424+0000","last_unstale":"2023-07-14T19:23:39.284311+0000","last_undegraded":"2023-07-14T19:23:39.284311+0000","last_fullsized":"2023-07-14T19:23:39.284311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:49.753010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515589+0000","last_change":"2023-07-14T19:23:38.296954+0000","last_active":"2023-07-14T19:23:39.515589+0000","last_peered":"2023-07-14T19:23:39.515589+0000","last_clean":"2023-07-14T19:23:39.515589+0000","last_became_active":"2023-07-14T19:23:38.296143+0000","last_became_peered":"2023-07-14T19:23:38.296143+0000","last_unstale":"2023-07-14T19:23:39.515589+0000","last_undegraded":"2023-07-14T19:23:39.515589+0000","last_fullsized":"2023-07-14T19:23:39.515589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:58:27.181508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296982+0000","last_change":"2023-07-14T19:23:37.460543+0000","last_active":"2023-07-14T19:23:42.296982+0000","last_peered":"2023-07-14T19:23:42.296982+0000","last_clean":"2023-07-14T19:23:42.296982+0000","last_became_active":"2023-07-14T19:23:37.460230+0000","last_became_peered":"2023-07-14T19:23:37.460230+0000","last_unstale":"2023-07-14T19:23:42.296982+0000","last_undegraded":"2023-07-14T19:23:42.296982+0000","last_fullsized":"2023-07-14T19:23:42.296982+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:17:14.842081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284020+0000","last_change":"2023-07-14T19:23:38.302414+0000","last_active":"2023-07-14T19:23:39.284020+0000","last_peered":"2023-07-14T19:23:39.284020+0000","last_clean":"2023-07-14T19:23:39.284020+0000","last_became_active":"2023-07-14T19:23:38.302089+0000","last_became_peered":"2023-07-14T19:23:38.302089+0000","last_unstale":"2023-07-14T19:23:39.284020+0000","last_undegraded":"2023-07-14T19:23:39.284020+0000","last_fullsized":"2023-07-14T19:23:39.284020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:09:15.082408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515110+0000","last_change":"2023-07-14T19:23:37.462245+0000","last_active":"2023-07-14T19:23:39.515110+0000","last_peered":"2023-07-14T19:23:39.515110+0000","last_clean":"2023-07-14T19:23:39.515110+0000","last_became_active":"2023-07-14T19:23:37.461911+0000","last_became_peered":"2023-07-14T19:23:37.461911+0000","last_unstale":"2023-07-14T19:23:39.515110+0000","last_undegraded":"2023-07-14T19:23:39.515110+0000","last_fullsized":"2023-07-14T19:23:39.515110+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:53.399246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":337,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515010+0000","last_change":"2023-07-14T19:22:39.254277+0000","last_active":"2023-07-14T19:23:39.515010+0000","last_peered":"2023-07-14T19:23:39.515010+0000","last_clean":"2023-07-14T19:23:39.515010+0000","last_became_active":"2023-07-14T19:22:39.253670+0000","last_became_peered":"2023-07-14T19:22:39.253670+0000","last_unstale":"2023-07-14T19:23:39.515010+0000","last_undegraded":"2023-07-14T19:23:39.515010+0000","last_fullsized":"2023-07-14T19:23:39.515010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:19:52.229492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:19:52.229492+0000","last_clean_scrub_stamp":"2023-07-14T19:19:52.229492+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:22.025952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514166+0000","last_change":"2023-07-14T19:23:38.296671+0000","last_active":"2023-07-14T19:23:39.514166+0000","last_peered":"2023-07-14T19:23:39.514166+0000","last_clean":"2023-07-14T19:23:39.514166+0000","last_became_active":"2023-07-14T19:23:38.296175+0000","last_became_peered":"2023-07-14T19:23:38.296175+0000","last_unstale":"2023-07-14T19:23:39.514166+0000","last_undegraded":"2023-07-14T19:23:39.514166+0000","last_fullsized":"2023-07-14T19:23:39.514166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:35:01.140553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295431+0000","last_change":"2023-07-14T19:23:37.459269+0000","last_active":"2023-07-14T19:23:42.295431+0000","last_peered":"2023-07-14T19:23:42.295431+0000","last_clean":"2023-07-14T19:23:42.295431+0000","last_became_active":"2023-07-14T19:23:37.458999+0000","last_became_peered":"2023-07-14T19:23:37.458999+0000","last_unstale":"2023-07-14T19:23:42.295431+0000","last_undegraded":"2023-07-14T19:23:42.295431+0000","last_fullsized":"2023-07-14T19:23:42.295431+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:45.895670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"80'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475684+0000","last_change":"2023-07-14T19:23:37.463425+0000","last_active":"2023-07-14T19:23:41.475684+0000","last_peered":"2023-07-14T19:23:41.475684+0000","last_clean":"2023-07-14T19:23:41.475684+0000","last_became_active":"2023-07-14T19:23:37.463005+0000","last_became_peered":"2023-07-14T19:23:37.463005+0000","last_unstale":"2023-07-14T19:23:41.475684+0000","last_undegraded":"2023-07-14T19:23:41.475684+0000","last_fullsized":"2023-07-14T19:23:41.475684+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:22:15.482538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515091+0000","last_change":"2023-07-14T19:23:38.295556+0000","last_active":"2023-07-14T19:23:39.515091+0000","last_peered":"2023-07-14T19:23:39.515091+0000","last_clean":"2023-07-14T19:23:39.515091+0000","last_became_active":"2023-07-14T19:23:38.295233+0000","last_became_peered":"2023-07-14T19:23:38.295233+0000","last_unstale":"2023-07-14T19:23:39.515091+0000","last_undegraded":"2023-07-14T19:23:39.515091+0000","last_fullsized":"2023-07-14T19:23:39.515091+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:29.024672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515774+0000","last_change":"2023-07-14T19:23:37.464752+0000","last_active":"2023-07-14T19:23:39.515774+0000","last_peered":"2023-07-14T19:23:39.515774+0000","last_clean":"2023-07-14T19:23:39.515774+0000","last_became_active":"2023-07-14T19:23:37.464451+0000","last_became_peered":"2023-07-14T19:23:37.464451+0000","last_unstale":"2023-07-14T19:23:39.515774+0000","last_undegraded":"2023-07-14T19:23:39.515774+0000","last_fullsized":"2023-07-14T19:23:39.515774+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:30:13.403469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284070+0000","last_change":"2023-07-14T19:23:38.298475+0000","last_active":"2023-07-14T19:23:39.284070+0000","last_peered":"2023-07-14T19:23:39.284070+0000","last_clean":"2023-07-14T19:23:39.284070+0000","last_became_active":"2023-07-14T19:23:38.298141+0000","last_became_peered":"2023-07-14T19:23:38.298141+0000","last_unstale":"2023-07-14T19:23:39.284070+0000","last_undegraded":"2023-07-14T19:23:39.284070+0000","last_fullsized":"2023-07-14T19:23:39.284070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:02.834103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514928+0000","last_change":"2023-07-14T19:23:37.457182+0000","last_active":"2023-07-14T19:23:39.514928+0000","last_peered":"2023-07-14T19:23:39.514928+0000","last_clean":"2023-07-14T19:23:39.514928+0000","last_became_active":"2023-07-14T19:23:37.456825+0000","last_became_peered":"2023-07-14T19:23:37.456825+0000","last_unstale":"2023-07-14T19:23:39.514928+0000","last_undegraded":"2023-07-14T19:23:39.514928+0000","last_fullsized":"2023-07-14T19:23:39.514928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:03.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514703+0000","last_change":"2023-07-14T19:23:38.303732+0000","last_active":"2023-07-14T19:23:39.514703+0000","last_peered":"2023-07-14T19:23:39.514703+0000","last_clean":"2023-07-14T19:23:39.514703+0000","last_became_active":"2023-07-14T19:23:38.303296+0000","last_became_peered":"2023-07-14T19:23:38.303296+0000","last_unstale":"2023-07-14T19:23:39.514703+0000","last_undegraded":"2023-07-14T19:23:39.514703+0000","last_fullsized":"2023-07-14T19:23:39.514703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:55:39.991626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475191+0000","last_change":"2023-07-14T19:23:37.461183+0000","last_active":"2023-07-14T19:23:41.475191+0000","last_peered":"2023-07-14T19:23:41.475191+0000","last_clean":"2023-07-14T19:23:41.475191+0000","last_became_active":"2023-07-14T19:23:37.460876+0000","last_became_peered":"2023-07-14T19:23:37.460876+0000","last_unstale":"2023-07-14T19:23:41.475191+0000","last_undegraded":"2023-07-14T19:23:41.475191+0000","last_fullsized":"2023-07-14T19:23:41.475191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:31.746901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286421+0000","last_change":"2023-07-14T19:23:38.292032+0000","last_active":"2023-07-14T19:23:39.286421+0000","last_peered":"2023-07-14T19:23:39.286421+0000","last_clean":"2023-07-14T19:23:39.286421+0000","last_became_active":"2023-07-14T19:23:38.291721+0000","last_became_peered":"2023-07-14T19:23:38.291721+0000","last_unstale":"2023-07-14T19:23:39.286421+0000","last_undegraded":"2023-07-14T19:23:39.286421+0000","last_fullsized":"2023-07-14T19:23:39.286421+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:58.334225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283442+0000","last_change":"2023-07-14T19:23:37.465597+0000","last_active":"2023-07-14T19:23:39.283442+0000","last_peered":"2023-07-14T19:23:39.283442+0000","last_clean":"2023-07-14T19:23:39.283442+0000","last_became_active":"2023-07-14T19:23:37.465233+0000","last_became_peered":"2023-07-14T19:23:37.465233+0000","last_unstale":"2023-07-14T19:23:39.283442+0000","last_undegraded":"2023-07-14T19:23:39.283442+0000","last_fullsized":"2023-07-14T19:23:39.283442+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:55.848783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472581+0000","last_change":"2023-07-14T19:23:38.295185+0000","last_active":"2023-07-14T19:23:39.472581+0000","last_peered":"2023-07-14T19:23:39.472581+0000","last_clean":"2023-07-14T19:23:39.472581+0000","last_became_active":"2023-07-14T19:23:38.294855+0000","last_became_peered":"2023-07-14T19:23:38.294855+0000","last_unstale":"2023-07-14T19:23:39.472581+0000","last_undegraded":"2023-07-14T19:23:39.472581+0000","last_fullsized":"2023-07-14T19:23:39.472581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:48.045146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:40.262742+0000","last_change":"2023-07-14T19:23:37.463517+0000","last_active":"2023-07-14T19:23:40.262742+0000","last_peered":"2023-07-14T19:23:40.262742+0000","last_clean":"2023-07-14T19:23:40.262742+0000","last_became_active":"2023-07-14T19:23:37.463158+0000","last_became_peered":"2023-07-14T19:23:37.463158+0000","last_unstale":"2023-07-14T19:23:40.262742+0000","last_undegraded":"2023-07-14T19:23:40.262742+0000","last_fullsized":"2023-07-14T19:23:40.262742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:01.142000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283721+0000","last_change":"2023-07-14T19:23:38.305375+0000","last_active":"2023-07-14T19:23:39.283721+0000","last_peered":"2023-07-14T19:23:39.283721+0000","last_clean":"2023-07-14T19:23:39.283721+0000","last_became_active":"2023-07-14T19:23:38.305084+0000","last_became_peered":"2023-07-14T19:23:38.305084+0000","last_unstale":"2023-07-14T19:23:39.283721+0000","last_undegraded":"2023-07-14T19:23:39.283721+0000","last_fullsized":"2023-07-14T19:23:39.283721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:08:51.652651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474944+0000","last_change":"2023-07-14T19:23:37.458672+0000","last_active":"2023-07-14T19:23:41.474944+0000","last_peered":"2023-07-14T19:23:41.474944+0000","last_clean":"2023-07-14T19:23:41.474944+0000","last_became_active":"2023-07-14T19:23:37.458359+0000","last_became_peered":"2023-07-14T19:23:37.458359+0000","last_unstale":"2023-07-14T19:23:41.474944+0000","last_undegraded":"2023-07-14T19:23:41.474944+0000","last_fullsized":"2023-07-14T19:23:41.474944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:00:25.922910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515725+0000","last_change":"2023-07-14T19:23:38.301741+0000","last_active":"2023-07-14T19:23:39.515725+0000","last_peered":"2023-07-14T19:23:39.515725+0000","last_clean":"2023-07-14T19:23:39.515725+0000","last_became_active":"2023-07-14T19:23:38.301430+0000","last_became_peered":"2023-07-14T19:23:38.301430+0000","last_unstale":"2023-07-14T19:23:39.515725+0000","last_undegraded":"2023-07-14T19:23:39.515725+0000","last_fullsized":"2023-07-14T19:23:39.515725+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:13.088226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296558+0000","last_change":"2023-07-14T19:23:37.461764+0000","last_active":"2023-07-14T19:23:42.296558+0000","last_peered":"2023-07-14T19:23:42.296558+0000","last_clean":"2023-07-14T19:23:42.296558+0000","last_became_active":"2023-07-14T19:23:37.460910+0000","last_became_peered":"2023-07-14T19:23:37.460910+0000","last_unstale":"2023-07-14T19:23:42.296558+0000","last_undegraded":"2023-07-14T19:23:42.296558+0000","last_fullsized":"2023-07-14T19:23:42.296558+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:39.478046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515703+0000","last_change":"2023-07-14T19:23:38.296214+0000","last_active":"2023-07-14T19:23:39.515703+0000","last_peered":"2023-07-14T19:23:39.515703+0000","last_clean":"2023-07-14T19:23:39.515703+0000","last_became_active":"2023-07-14T19:23:38.295148+0000","last_became_peered":"2023-07-14T19:23:38.295148+0000","last_unstale":"2023-07-14T19:23:39.515703+0000","last_undegraded":"2023-07-14T19:23:39.515703+0000","last_fullsized":"2023-07-14T19:23:39.515703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:30.117202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515939+0000","last_change":"2023-07-14T19:23:37.458245+0000","last_active":"2023-07-14T19:23:39.515939+0000","last_peered":"2023-07-14T19:23:39.515939+0000","last_clean":"2023-07-14T19:23:39.515939+0000","last_became_active":"2023-07-14T19:23:37.457784+0000","last_became_peered":"2023-07-14T19:23:37.457784+0000","last_unstale":"2023-07-14T19:23:39.515939+0000","last_undegraded":"2023-07-14T19:23:39.515939+0000","last_fullsized":"2023-07-14T19:23:39.515939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:47:00.389865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515321+0000","last_change":"2023-07-14T19:23:38.299823+0000","last_active":"2023-07-14T19:23:39.515321+0000","last_peered":"2023-07-14T19:23:39.515321+0000","last_clean":"2023-07-14T19:23:39.515321+0000","last_became_active":"2023-07-14T19:23:38.299496+0000","last_became_peered":"2023-07-14T19:23:38.299496+0000","last_unstale":"2023-07-14T19:23:39.515321+0000","last_undegraded":"2023-07-14T19:23:39.515321+0000","last_fullsized":"2023-07-14T19:23:39.515321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:23:40.353266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514282+0000","last_change":"2023-07-14T19:23:37.463439+0000","last_active":"2023-07-14T19:23:39.514282+0000","last_peered":"2023-07-14T19:23:39.514282+0000","last_clean":"2023-07-14T19:23:39.514282+0000","last_became_active":"2023-07-14T19:23:37.463100+0000","last_became_peered":"2023-07-14T19:23:37.463100+0000","last_unstale":"2023-07-14T19:23:39.514282+0000","last_undegraded":"2023-07-14T19:23:39.514282+0000","last_fullsized":"2023-07-14T19:23:39.514282+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:56.140535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515366+0000","last_change":"2023-07-14T19:23:38.296265+0000","last_active":"2023-07-14T19:23:39.515366+0000","last_peered":"2023-07-14T19:23:39.515366+0000","last_clean":"2023-07-14T19:23:39.515366+0000","last_became_active":"2023-07-14T19:23:38.295918+0000","last_became_peered":"2023-07-14T19:23:38.295918+0000","last_unstale":"2023-07-14T19:23:39.515366+0000","last_undegraded":"2023-07-14T19:23:39.515366+0000","last_fullsized":"2023-07-14T19:23:39.515366+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:38.769993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472522+0000","last_change":"2023-07-14T19:23:38.300200+0000","last_active":"2023-07-14T19:23:39.472522+0000","last_peered":"2023-07-14T19:23:39.472522+0000","last_clean":"2023-07-14T19:23:39.472522+0000","last_became_active":"2023-07-14T19:23:38.299979+0000","last_became_peered":"2023-07-14T19:23:38.299979+0000","last_unstale":"2023-07-14T19:23:39.472522+0000","last_undegraded":"2023-07-14T19:23:39.472522+0000","last_fullsized":"2023-07-14T19:23:39.472522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:00.789219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514842+0000","last_change":"2023-07-14T19:23:38.300696+0000","last_active":"2023-07-14T19:23:39.514842+0000","last_peered":"2023-07-14T19:23:39.514842+0000","last_clean":"2023-07-14T19:23:39.514842+0000","last_became_active":"2023-07-14T19:23:38.300377+0000","last_became_peered":"2023-07-14T19:23:38.300377+0000","last_unstale":"2023-07-14T19:23:39.514842+0000","last_undegraded":"2023-07-14T19:23:39.514842+0000","last_fullsized":"2023-07-14T19:23:39.514842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:39:05.440224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514814+0000","last_change":"2023-07-14T19:23:38.298616+0000","last_active":"2023-07-14T19:23:39.514814+0000","last_peered":"2023-07-14T19:23:39.514814+0000","last_clean":"2023-07-14T19:23:39.514814+0000","last_became_active":"2023-07-14T19:23:38.298313+0000","last_became_peered":"2023-07-14T19:23:38.298313+0000","last_unstale":"2023-07-14T19:23:39.514814+0000","last_undegraded":"2023-07-14T19:23:39.514814+0000","last_fullsized":"2023-07-14T19:23:39.514814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:46:49.950796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514982+0000","last_change":"2023-07-14T19:23:38.297185+0000","last_active":"2023-07-14T19:23:39.514982+0000","last_peered":"2023-07-14T19:23:39.514982+0000","last_clean":"2023-07-14T19:23:39.514982+0000","last_became_active":"2023-07-14T19:23:38.296883+0000","last_became_peered":"2023-07-14T19:23:38.296883+0000","last_unstale":"2023-07-14T19:23:39.514982+0000","last_undegraded":"2023-07-14T19:23:39.514982+0000","last_fullsized":"2023-07-14T19:23:39.514982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:08:27.715078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514907+0000","last_change":"2023-07-14T19:23:38.296636+0000","last_active":"2023-07-14T19:23:39.514907+0000","last_peered":"2023-07-14T19:23:39.514907+0000","last_clean":"2023-07-14T19:23:39.514907+0000","last_became_active":"2023-07-14T19:23:38.296289+0000","last_became_peered":"2023-07-14T19:23:38.296289+0000","last_unstale":"2023-07-14T19:23:39.514907+0000","last_undegraded":"2023-07-14T19:23:39.514907+0000","last_fullsized":"2023-07-14T19:23:39.514907+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:18:30.937689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472626+0000","last_change":"2023-07-14T19:23:38.300340+0000","last_active":"2023-07-14T19:23:39.472626+0000","last_peered":"2023-07-14T19:23:39.472626+0000","last_clean":"2023-07-14T19:23:39.472626+0000","last_became_active":"2023-07-14T19:23:38.299520+0000","last_became_peered":"2023-07-14T19:23:38.299520+0000","last_unstale":"2023-07-14T19:23:39.472626+0000","last_undegraded":"2023-07-14T19:23:39.472626+0000","last_fullsized":"2023-07-14T19:23:39.472626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:03:47.657130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515538+0000","last_change":"2023-07-14T19:23:38.294136+0000","last_active":"2023-07-14T19:23:39.515538+0000","last_peered":"2023-07-14T19:23:39.515538+0000","last_clean":"2023-07-14T19:23:39.515538+0000","last_became_active":"2023-07-14T19:23:38.293785+0000","last_became_peered":"2023-07-14T19:23:38.293785+0000","last_unstale":"2023-07-14T19:23:39.515538+0000","last_undegraded":"2023-07-14T19:23:39.515538+0000","last_fullsized":"2023-07-14T19:23:39.515538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:21.794486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284002+0000","last_change":"2023-07-14T19:23:38.305648+0000","last_active":"2023-07-14T19:23:39.284002+0000","last_peered":"2023-07-14T19:23:39.284002+0000","last_clean":"2023-07-14T19:23:39.284002+0000","last_became_active":"2023-07-14T19:23:38.305314+0000","last_became_peered":"2023-07-14T19:23:38.305314+0000","last_unstale":"2023-07-14T19:23:39.284002+0000","last_undegraded":"2023-07-14T19:23:39.284002+0000","last_fullsized":"2023-07-14T19:23:39.284002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:01:05.266523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515131+0000","last_change":"2023-07-14T19:23:38.298189+0000","last_active":"2023-07-14T19:23:39.515131+0000","last_peered":"2023-07-14T19:23:39.515131+0000","last_clean":"2023-07-14T19:23:39.515131+0000","last_became_active":"2023-07-14T19:23:38.297781+0000","last_became_peered":"2023-07-14T19:23:38.297781+0000","last_unstale":"2023-07-14T19:23:39.515131+0000","last_undegraded":"2023-07-14T19:23:39.515131+0000","last_fullsized":"2023-07-14T19:23:39.515131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:49:59.706932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283450+0000","last_change":"2023-07-14T19:23:38.293567+0000","last_active":"2023-07-14T19:23:39.283450+0000","last_peered":"2023-07-14T19:23:39.283450+0000","last_clean":"2023-07-14T19:23:39.283450+0000","last_became_active":"2023-07-14T19:23:38.293222+0000","last_became_peered":"2023-07-14T19:23:38.293222+0000","last_unstale":"2023-07-14T19:23:39.283450+0000","last_undegraded":"2023-07-14T19:23:39.283450+0000","last_fullsized":"2023-07-14T19:23:39.283450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:28.545521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515396+0000","last_change":"2023-07-14T19:23:38.294168+0000","last_active":"2023-07-14T19:23:39.515396+0000","last_peered":"2023-07-14T19:23:39.515396+0000","last_clean":"2023-07-14T19:23:39.515396+0000","last_became_active":"2023-07-14T19:23:38.293665+0000","last_became_peered":"2023-07-14T19:23:38.293665+0000","last_unstale":"2023-07-14T19:23:39.515396+0000","last_undegraded":"2023-07-14T19:23:39.515396+0000","last_fullsized":"2023-07-14T19:23:39.515396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:49.466175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515479+0000","last_change":"2023-07-14T19:23:38.296535+0000","last_active":"2023-07-14T19:23:39.515479+0000","last_peered":"2023-07-14T19:23:39.515479+0000","last_clean":"2023-07-14T19:23:39.515479+0000","last_became_active":"2023-07-14T19:23:38.296189+0000","last_became_peered":"2023-07-14T19:23:38.296189+0000","last_unstale":"2023-07-14T19:23:39.515479+0000","last_undegraded":"2023-07-14T19:23:39.515479+0000","last_fullsized":"2023-07-14T19:23:39.515479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:36:54.183312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472559+0000","last_change":"2023-07-14T19:23:38.296867+0000","last_active":"2023-07-14T19:23:39.472559+0000","last_peered":"2023-07-14T19:23:39.472559+0000","last_clean":"2023-07-14T19:23:39.472559+0000","last_became_active":"2023-07-14T19:23:38.296560+0000","last_became_peered":"2023-07-14T19:23:38.296560+0000","last_unstale":"2023-07-14T19:23:39.472559+0000","last_undegraded":"2023-07-14T19:23:39.472559+0000","last_fullsized":"2023-07-14T19:23:39.472559+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:48:16.893808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":39087,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":28,"num_read_kb":39,"num_write":111,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":100763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547212,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":860160,"data_stored":413577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":10,"up_from":68,"seq":292057776143,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28512,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716640,"statfs":{"total":95995035648,"available":95965839360,"internally_reserved":0,"allocated":1531904,"data_stored":1041504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40799999999999997}]},{"osd":2,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.61399999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.53800000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.498},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.55300000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.65900000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.59299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.72599999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.68200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":9,"up_from":63,"seq":270582939667,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":414159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.307},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.6080000000000001,"5min":2.6080000000000001,"15min":2.6080000000000001},"last":0.41799999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.657,"5min":2.657,"15min":2.657},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":0.52400000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":2.7589999999999999,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.497},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.59399999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.6699999999999999,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.9740000000000002,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.53500000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.69099999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.7309999999999999,"5min":2.7309999999999999,"15min":2.7309999999999999},"last":0.66500000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":0.70899999999999996}]}]},{"osd":8,"up_from":56,"seq":240518168600,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":413984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.443},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.72599999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.53600000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.75}]},{"osd":5,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.65700000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51900000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.628},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76500000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:23:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.56200000000000006}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":7,"up_from":48,"seq":206158430235,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1466368,"data_stored":1005101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73099999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.82299999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.43099999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42499999999999999,"15min":0.42099999999999999},"min":{"1min":0.32900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78600000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.45800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.83699999999999997}]},{"osd":4,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.502,"15min":0.49399999999999999},"min":{"1min":0.36299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.69399999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.621,"15min":0.624},"min":{"1min":0.47699999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66600000000000004,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.76100000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.51300000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66400000000000003},"min":{"1min":0.41199999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.77500000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.753}]},{"osd":9,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.90900000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.81599999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":6,"up_from":43,"seq":184683593761,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":413667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.30299999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60999999999999999,"5min":0.62,"15min":0.62},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.72799999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.35399999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.39500000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.61699999999999999,"15min":0.60899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.62},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.63700000000000001,"15min":0.621},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52000000000000002,"15min":0.51300000000000001},"min":{"1min":0.30599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.54300000000000004,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.875,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.40999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.66300000000000003,"15min":0.64700000000000002},"min":{"1min":0.45100000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.60499999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.67200000000000004,"15min":0.66000000000000003},"min":{"1min":0.46700000000000003,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61499999999999999,"15min":0.61199999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64100000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.57099999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.71399999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.44600000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":432233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53200000000000003,"15min":0.52200000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83999999999999997,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.46600000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53000000000000003,"15min":0.51700000000000002},"min":{"1min":0.42199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.85899999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.54700000000000004,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.878,"5min":1.028,"15min":1.028},"last":0.68000000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.40300000000000002},"min":{"1min":0.27600000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.433,"15min":0.42499999999999999},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.58099999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.873},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.89400000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.61699999999999999,"15min":0.60599999999999998},"min":{"1min":0.503,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65900000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.73099999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.95599999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:23:15 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.97899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":872448,"data_stored":414271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.52500000000000002,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.84599999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.51400000000000001,"15min":0.46800000000000003},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.51800000000000002,"15min":0.5},"min":{"1min":0.36599999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.54200000000000004,"15min":0.52300000000000002},"min":{"1min":0.44400000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48799999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.432,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.371},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.433},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.42699999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59099999999999997,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.58699999999999997,"15min":0.58199999999999996},"min":{"1min":0.41199999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61499999999999999,"15min":0.60599999999999998},"min":{"1min":0.42299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.623}]},{"osd":6,"last update":"Fri Jul 14 19:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.46999999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67600000000000005,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.434,"15min":0.434},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.73099999999999998,"15min":0.72699999999999998},"min":{"1min":0.55900000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.58599999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.56299999999999994}]},{"osd":9,"last update":"Fri Jul 14 19:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.60499999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.67800000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":434198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.499,"15min":0.48999999999999999},"min":{"1min":0.35299999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.52400000000000002,"15min":0.5},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.56399999999999995,"15min":0.55000000000000004},"min":{"1min":0.45800000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.50700000000000001},"min":{"1min":0.438,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72599999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69299999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.69699999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.44,"15min":0.42799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.624},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.46899999999999997,"15min":0.46200000000000002},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.80300000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.47599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.434,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82799999999999996,"5min":3.6579999999999999,"15min":3.6579999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.79900000000000004,"15min":0.82899999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.79500000000000004,"5min":3.8769999999999998,"15min":3.8769999999999998},"last":0.45900000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.82999999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.78800000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.75},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.879}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":3,"up_from":24,"seq":103079215148,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":868352,"data_stored":413681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.34899999999999998},"min":{"1min":0.30399999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.432},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.39000000000000001,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.33900000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.54200000000000004,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.436,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.48999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.57899999999999996,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.72499999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.57699999999999996,"15min":0.55500000000000005},"min":{"1min":0.47299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.625,"15min":0.59999999999999998},"min":{"1min":0.41499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.95799999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.51100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.54900000000000004,"15min":0.53500000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.752}]},{"osd":6,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.78200000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.443,"5min":0.436,"15min":0.436},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.87},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.73099999999999998,"15min":0.71699999999999997},"min":{"1min":0.54300000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":1.0009999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.54800000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.85199999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.92100000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986215,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":864256,"data_stored":413599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.34499999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.44,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73999999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.46200000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.36799999999999999,"15min":0.33800000000000002},"min":{"1min":0.29499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.78200000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.39700000000000002,"15min":0.377},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.81599999999999995,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.54500000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.755},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.51900000000000002},"min":{"1min":0.48799999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88900000000000001,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.40000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84799999999999998,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.629},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.47099999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.48599999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.59199999999999997,"15min":0.57199999999999995},"min":{"1min":0.53100000000000003,"5min":0.372,"15min":0.372},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.85199999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.57199999999999995,"15min":0.56899999999999995},"min":{"1min":0.42299999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.94799999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.53900000000000003,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.496},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.442,"5min":0.377,"15min":0.377},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.70999999999999996}]},{"osd":8,"last update":"Fri Jul 14 19:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69099999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.77900000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.72699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":5,"up_from":36,"seq":154618822692,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1478656,"data_stored":1005042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.40500000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55100000000000005}]},{"osd":1,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.755,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.29099999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.39300000000000002},"min":{"1min":0.29099999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.61699999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.46400000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.57099999999999995,"15min":0.56299999999999994},"min":{"1min":0.42499999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.72699999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.65600000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.79200000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.51300000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64600000000000002,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.82399999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.86199999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.84799999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.89700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":18929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":37929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":37767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:23:46.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T19:23:46.212 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.209+0000 7f1b5dacb700 1 -- 172.21.15.53:0/3821939775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 msgr2=0x7f1b58105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:46.212 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.209+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/3821939775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 0x7f1b58105380 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009990 tx=0x7f1b4c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.212 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.209+0000 7f1b5dacb700 1 -- 172.21.15.53:0/3821939775 shutdown_connections 2023-07-14T19:23:46.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.209+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/3821939775 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b581058c0 0x7f1b58107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.209+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/3821939775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b58100680 0x7f1b58102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.209+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/3821939775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 0x7f1b58105380 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.209+0000 7f1b5dacb700 1 -- 172.21.15.53:0/3821939775 >> 172.21.15.53:0/3821939775 conn(0x7f1b580fa100 msgr2=0x7f1b580fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:46.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.209+0000 7f1b5dacb700 1 -- 172.21.15.53:0/3821939775 shutdown_connections 2023-07-14T19:23:46.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.209+0000 7f1b5dacb700 1 -- 172.21.15.53:0/3821939775 wait complete. 2023-07-14T19:23:46.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.210+0000 7f1b5dacb700 1 Processor -- start 2023-07-14T19:23:46.215 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.210+0000 7f1b5dacb700 1 -- start start 2023-07-14T19:23:46.215 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.210+0000 7f1b5dacb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b58100680 0x7f1b581958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:46.216 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.210+0000 7f1b5dacb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 0x7f1b58195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:46.216 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.210+0000 7f1b5dacb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b581058c0 0x7f1b5819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:46.216 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.210+0000 7f1b5dacb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5819d430 con 0x7f1b58102fa0 2023-07-14T19:23:46.216 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.210+0000 7f1b5dacb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5819d570 con 0x7f1b581058c0 2023-07-14T19:23:46.216 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.210+0000 7f1b5dacb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5819d700 con 0x7f1b58100680 2023-07-14T19:23:46.216 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b56ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 0x7f1b58195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:46.217 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b56ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 0x7f1b58195df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50408/0 (socket says 172.21.15.53:50408) 2023-07-14T19:23:46.217 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b56ffd700 1 -- 172.21.15.53:0/2215395526 learned_addr learned my addr 172.21.15.53:0/2215395526 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:46.217 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b577fe700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b58100680 0x7f1b581958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:46.217 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b57fff700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b581058c0 0x7f1b5819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:46.217 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b56ffd700 1 -- 172.21.15.53:0/2215395526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b58100680 msgr2=0x7f1b581958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:46.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b56ffd700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b58100680 0x7f1b581958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b56ffd700 1 -- 172.21.15.53:0/2215395526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b581058c0 msgr2=0x7f1b5819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:46.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b56ffd700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b581058c0 0x7f1b5819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b56ffd700 1 -- 172.21.15.53:0/2215395526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c0095f0 con 0x7f1b58102fa0 2023-07-14T19:23:46.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b57fff700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b581058c0 0x7f1b5819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:23:46.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b56ffd700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 0x7f1b58195df0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0059d0 tx=0x7f1b4c0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:46.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b54ff9700 1 -- 172.21.15.53:0/2215395526 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c01d070 con 0x7f1b58102fa0 2023-07-14T19:23:46.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b54ff9700 1 -- 172.21.15.53:0/2215395526 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b4c00fc40 con 0x7f1b58102fa0 2023-07-14T19:23:46.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5819d980 con 0x7f1b58102fa0 2023-07-14T19:23:46.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b54ff9700 1 -- 172.21.15.53:0/2215395526 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c017650 con 0x7f1b58102fa0 2023-07-14T19:23:46.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.211+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5819de00 con 0x7f1b58102fa0 2023-07-14T19:23:46.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.212+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b580fbc70 con 0x7f1b58102fa0 2023-07-14T19:23:46.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.214+0000 7f1b54ff9700 1 -- 172.21.15.53:0/2215395526 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1b4c022470 con 0x7f1b58102fa0 2023-07-14T19:23:46.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.214+0000 7f1b54ff9700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1b400a1320 0x7f1b400a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:46.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.214+0000 7f1b577fe700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1b400a1320 0x7f1b400a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:46.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.214+0000 7f1b54ff9700 1 -- 172.21.15.53:0/2215395526 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1b4c0ce690 con 0x7f1b58102fa0 2023-07-14T19:23:46.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.215+0000 7f1b577fe700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1b400a1320 0x7f1b400a37d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009c60 tx=0x7f1b48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:46.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.217+0000 7f1b54ff9700 1 -- 172.21.15.53:0/2215395526 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1b4c097dc0 con 0x7f1b58102fa0 2023-07-14T19:23:46.420 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.417+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b58060a80 con 0x7f1b400a1320 2023-07-14T19:23:46.432 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.429+0000 7f1b54ff9700 1 -- 172.21.15.53:0/2215395526 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335987 (secure 0 0 0) 0x7f1b58060a80 con 0x7f1b400a1320 2023-07-14T19:23:46.437 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:46.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1b400a1320 msgr2=0x7f1b400a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:46.440 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1b400a1320 0x7f1b400a37d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009c60 tx=0x7f1b48009260 comp rx=0 tx=0).stop 2023-07-14T19:23:46.440 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 msgr2=0x7f1b58195df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:46.440 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 0x7f1b58195df0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0059d0 tx=0x7f1b4c0047a0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.440 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 shutdown_connections 2023-07-14T19:23:46.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1b400a1320 0x7f1b400a37d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b581058c0 0x7f1b5819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1b58100680 0x7f1b581958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 --2- 172.21.15.53:0/2215395526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b58102fa0 0x7f1b58195df0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:46.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 >> 172.21.15.53:0/2215395526 conn(0x7f1b580fa100 msgr2=0x7f1b581012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:46.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 shutdown_connections 2023-07-14T19:23:46.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:46.433+0000 7f1b5dacb700 1 -- 172.21.15.53:0/2215395526 wait complete. 2023-07-14T19:23:46.443 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2023-07-14T19:23:46.447 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2023-07-14T19:23:45.315763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":629578,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":164,"num_read_kb":158,"num_write":293,"num_write_kb":3003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":277,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336276,"kb_used_data":12056,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605548,"statfs":{"total":1151940427776,"available":1151596081152,"internally_reserved":0,"allocated":12345344,"data_stored":6815016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39210,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":28,"num_read_kb":39,"num_write":112,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.870748"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472803+0000","last_change":"2023-07-14T19:23:38.301785+0000","last_active":"2023-07-14T19:23:39.472803+0000","last_peered":"2023-07-14T19:23:39.472803+0000","last_clean":"2023-07-14T19:23:39.472803+0000","last_became_active":"2023-07-14T19:23:38.301014+0000","last_became_peered":"2023-07-14T19:23:38.301014+0000","last_unstale":"2023-07-14T19:23:39.472803+0000","last_undegraded":"2023-07-14T19:23:39.472803+0000","last_fullsized":"2023-07-14T19:23:39.472803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:02.663606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514628+0000","last_change":"2023-07-14T19:23:38.299101+0000","last_active":"2023-07-14T19:23:39.514628+0000","last_peered":"2023-07-14T19:23:39.514628+0000","last_clean":"2023-07-14T19:23:39.514628+0000","last_became_active":"2023-07-14T19:23:38.298720+0000","last_became_peered":"2023-07-14T19:23:38.298720+0000","last_unstale":"2023-07-14T19:23:39.514628+0000","last_undegraded":"2023-07-14T19:23:39.514628+0000","last_fullsized":"2023-07-14T19:23:39.514628+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:05:31.873298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514167+0000","last_change":"2023-07-14T19:23:38.295335+0000","last_active":"2023-07-14T19:23:39.514167+0000","last_peered":"2023-07-14T19:23:39.514167+0000","last_clean":"2023-07-14T19:23:39.514167+0000","last_became_active":"2023-07-14T19:23:38.295031+0000","last_became_peered":"2023-07-14T19:23:38.295031+0000","last_unstale":"2023-07-14T19:23:39.514167+0000","last_undegraded":"2023-07-14T19:23:39.514167+0000","last_fullsized":"2023-07-14T19:23:39.514167+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:09.443353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515703+0000","last_change":"2023-07-14T19:23:38.295055+0000","last_active":"2023-07-14T19:23:39.515703+0000","last_peered":"2023-07-14T19:23:39.515703+0000","last_clean":"2023-07-14T19:23:39.515703+0000","last_became_active":"2023-07-14T19:23:38.294751+0000","last_became_peered":"2023-07-14T19:23:38.294751+0000","last_unstale":"2023-07-14T19:23:39.515703+0000","last_undegraded":"2023-07-14T19:23:39.515703+0000","last_fullsized":"2023-07-14T19:23:39.515703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:19.020190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286436+0000","last_change":"2023-07-14T19:23:38.294684+0000","last_active":"2023-07-14T19:23:39.286436+0000","last_peered":"2023-07-14T19:23:39.286436+0000","last_clean":"2023-07-14T19:23:39.286436+0000","last_became_active":"2023-07-14T19:23:38.294368+0000","last_became_peered":"2023-07-14T19:23:38.294368+0000","last_unstale":"2023-07-14T19:23:39.286436+0000","last_undegraded":"2023-07-14T19:23:39.286436+0000","last_fullsized":"2023-07-14T19:23:39.286436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:35.017660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283944+0000","last_change":"2023-07-14T19:23:38.301219+0000","last_active":"2023-07-14T19:23:39.283944+0000","last_peered":"2023-07-14T19:23:39.283944+0000","last_clean":"2023-07-14T19:23:39.283944+0000","last_became_active":"2023-07-14T19:23:38.300904+0000","last_became_peered":"2023-07-14T19:23:38.300904+0000","last_unstale":"2023-07-14T19:23:39.283944+0000","last_undegraded":"2023-07-14T19:23:39.283944+0000","last_fullsized":"2023-07-14T19:23:39.283944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:53:02.603728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515586+0000","last_change":"2023-07-14T19:23:38.292831+0000","last_active":"2023-07-14T19:23:39.515586+0000","last_peered":"2023-07-14T19:23:39.515586+0000","last_clean":"2023-07-14T19:23:39.515586+0000","last_became_active":"2023-07-14T19:23:38.292536+0000","last_became_peered":"2023-07-14T19:23:38.292536+0000","last_unstale":"2023-07-14T19:23:39.515586+0000","last_undegraded":"2023-07-14T19:23:39.515586+0000","last_fullsized":"2023-07-14T19:23:39.515586+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:49.316491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283393+0000","last_change":"2023-07-14T19:23:38.304940+0000","last_active":"2023-07-14T19:23:39.283393+0000","last_peered":"2023-07-14T19:23:39.283393+0000","last_clean":"2023-07-14T19:23:39.283393+0000","last_became_active":"2023-07-14T19:23:38.304609+0000","last_became_peered":"2023-07-14T19:23:38.304609+0000","last_unstale":"2023-07-14T19:23:39.283393+0000","last_undegraded":"2023-07-14T19:23:39.283393+0000","last_fullsized":"2023-07-14T19:23:39.283393+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:14.673980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515328+0000","last_change":"2023-07-14T19:23:38.300273+0000","last_active":"2023-07-14T19:23:39.515328+0000","last_peered":"2023-07-14T19:23:39.515328+0000","last_clean":"2023-07-14T19:23:39.515328+0000","last_became_active":"2023-07-14T19:23:38.299965+0000","last_became_peered":"2023-07-14T19:23:38.299965+0000","last_unstale":"2023-07-14T19:23:39.515328+0000","last_undegraded":"2023-07-14T19:23:39.515328+0000","last_fullsized":"2023-07-14T19:23:39.515328+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:26:21.737406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284243+0000","last_change":"2023-07-14T19:23:38.302757+0000","last_active":"2023-07-14T19:23:39.284243+0000","last_peered":"2023-07-14T19:23:39.284243+0000","last_clean":"2023-07-14T19:23:39.284243+0000","last_became_active":"2023-07-14T19:23:38.302409+0000","last_became_peered":"2023-07-14T19:23:38.302409+0000","last_unstale":"2023-07-14T19:23:39.284243+0000","last_undegraded":"2023-07-14T19:23:39.284243+0000","last_fullsized":"2023-07-14T19:23:39.284243+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:09:43.204311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515622+0000","last_change":"2023-07-14T19:23:38.296974+0000","last_active":"2023-07-14T19:23:39.515622+0000","last_peered":"2023-07-14T19:23:39.515622+0000","last_clean":"2023-07-14T19:23:39.515622+0000","last_became_active":"2023-07-14T19:23:38.296718+0000","last_became_peered":"2023-07-14T19:23:38.296718+0000","last_unstale":"2023-07-14T19:23:39.515622+0000","last_undegraded":"2023-07-14T19:23:39.515622+0000","last_fullsized":"2023-07-14T19:23:39.515622+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:19:13.680031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515564+0000","last_change":"2023-07-14T19:23:38.297575+0000","last_active":"2023-07-14T19:23:39.515564+0000","last_peered":"2023-07-14T19:23:39.515564+0000","last_clean":"2023-07-14T19:23:39.515564+0000","last_became_active":"2023-07-14T19:23:38.297292+0000","last_became_peered":"2023-07-14T19:23:38.297292+0000","last_unstale":"2023-07-14T19:23:39.515564+0000","last_undegraded":"2023-07-14T19:23:39.515564+0000","last_fullsized":"2023-07-14T19:23:39.515564+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:49.161960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515373+0000","last_change":"2023-07-14T19:23:38.295810+0000","last_active":"2023-07-14T19:23:39.515373+0000","last_peered":"2023-07-14T19:23:39.515373+0000","last_clean":"2023-07-14T19:23:39.515373+0000","last_became_active":"2023-07-14T19:23:38.295281+0000","last_became_peered":"2023-07-14T19:23:38.295281+0000","last_unstale":"2023-07-14T19:23:39.515373+0000","last_undegraded":"2023-07-14T19:23:39.515373+0000","last_fullsized":"2023-07-14T19:23:39.515373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:37.869421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514485+0000","last_change":"2023-07-14T19:23:38.299271+0000","last_active":"2023-07-14T19:23:39.514485+0000","last_peered":"2023-07-14T19:23:39.514485+0000","last_clean":"2023-07-14T19:23:39.514485+0000","last_became_active":"2023-07-14T19:23:38.298298+0000","last_became_peered":"2023-07-14T19:23:38.298298+0000","last_unstale":"2023-07-14T19:23:39.514485+0000","last_undegraded":"2023-07-14T19:23:39.514485+0000","last_fullsized":"2023-07-14T19:23:39.514485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:30.297961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514219+0000","last_change":"2023-07-14T19:23:38.296295+0000","last_active":"2023-07-14T19:23:39.514219+0000","last_peered":"2023-07-14T19:23:39.514219+0000","last_clean":"2023-07-14T19:23:39.514219+0000","last_became_active":"2023-07-14T19:23:38.296004+0000","last_became_peered":"2023-07-14T19:23:38.296004+0000","last_unstale":"2023-07-14T19:23:39.514219+0000","last_undegraded":"2023-07-14T19:23:39.514219+0000","last_fullsized":"2023-07-14T19:23:39.514219+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:34:50.891851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514438+0000","last_change":"2023-07-14T19:23:38.293292+0000","last_active":"2023-07-14T19:23:39.514438+0000","last_peered":"2023-07-14T19:23:39.514438+0000","last_clean":"2023-07-14T19:23:39.514438+0000","last_became_active":"2023-07-14T19:23:38.292980+0000","last_became_peered":"2023-07-14T19:23:38.292980+0000","last_unstale":"2023-07-14T19:23:39.514438+0000","last_undegraded":"2023-07-14T19:23:39.514438+0000","last_fullsized":"2023-07-14T19:23:39.514438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:22.060162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472781+0000","last_change":"2023-07-14T19:23:38.301950+0000","last_active":"2023-07-14T19:23:39.472781+0000","last_peered":"2023-07-14T19:23:39.472781+0000","last_clean":"2023-07-14T19:23:39.472781+0000","last_became_active":"2023-07-14T19:23:38.301416+0000","last_became_peered":"2023-07-14T19:23:38.301416+0000","last_unstale":"2023-07-14T19:23:39.472781+0000","last_undegraded":"2023-07-14T19:23:39.472781+0000","last_fullsized":"2023-07-14T19:23:39.472781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:03:11.369261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515639+0000","last_change":"2023-07-14T19:23:38.300108+0000","last_active":"2023-07-14T19:23:39.515639+0000","last_peered":"2023-07-14T19:23:39.515639+0000","last_clean":"2023-07-14T19:23:39.515639+0000","last_became_active":"2023-07-14T19:23:38.299773+0000","last_became_peered":"2023-07-14T19:23:38.299773+0000","last_unstale":"2023-07-14T19:23:39.515639+0000","last_undegraded":"2023-07-14T19:23:39.515639+0000","last_fullsized":"2023-07-14T19:23:39.515639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:59.607746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515238+0000","last_change":"2023-07-14T19:23:38.299124+0000","last_active":"2023-07-14T19:23:39.515238+0000","last_peered":"2023-07-14T19:23:39.515238+0000","last_clean":"2023-07-14T19:23:39.515238+0000","last_became_active":"2023-07-14T19:23:38.298744+0000","last_became_peered":"2023-07-14T19:23:38.298744+0000","last_unstale":"2023-07-14T19:23:39.515238+0000","last_undegraded":"2023-07-14T19:23:39.515238+0000","last_fullsized":"2023-07-14T19:23:39.515238+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:09.383666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515197+0000","last_change":"2023-07-14T19:23:37.466299+0000","last_active":"2023-07-14T19:23:39.515197+0000","last_peered":"2023-07-14T19:23:39.515197+0000","last_clean":"2023-07-14T19:23:39.515197+0000","last_became_active":"2023-07-14T19:23:37.465989+0000","last_became_peered":"2023-07-14T19:23:37.465989+0000","last_unstale":"2023-07-14T19:23:39.515197+0000","last_undegraded":"2023-07-14T19:23:39.515197+0000","last_fullsized":"2023-07-14T19:23:39.515197+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:28.346196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283813+0000","last_change":"2023-07-14T19:23:38.301743+0000","last_active":"2023-07-14T19:23:39.283813+0000","last_peered":"2023-07-14T19:23:39.283813+0000","last_clean":"2023-07-14T19:23:39.283813+0000","last_became_active":"2023-07-14T19:23:38.301419+0000","last_became_peered":"2023-07-14T19:23:38.301419+0000","last_unstale":"2023-07-14T19:23:39.283813+0000","last_undegraded":"2023-07-14T19:23:39.283813+0000","last_fullsized":"2023-07-14T19:23:39.283813+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:37.577409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283986+0000","last_change":"2023-07-14T19:23:37.459314+0000","last_active":"2023-07-14T19:23:39.283986+0000","last_peered":"2023-07-14T19:23:39.283986+0000","last_clean":"2023-07-14T19:23:39.283986+0000","last_became_active":"2023-07-14T19:23:37.458925+0000","last_became_peered":"2023-07-14T19:23:37.458925+0000","last_unstale":"2023-07-14T19:23:39.283986+0000","last_undegraded":"2023-07-14T19:23:39.283986+0000","last_fullsized":"2023-07-14T19:23:39.283986+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:59.317422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514675+0000","last_change":"2023-07-14T19:23:38.294432+0000","last_active":"2023-07-14T19:23:39.514675+0000","last_peered":"2023-07-14T19:23:39.514675+0000","last_clean":"2023-07-14T19:23:39.514675+0000","last_became_active":"2023-07-14T19:23:38.294118+0000","last_became_peered":"2023-07-14T19:23:38.294118+0000","last_unstale":"2023-07-14T19:23:39.514675+0000","last_undegraded":"2023-07-14T19:23:39.514675+0000","last_fullsized":"2023-07-14T19:23:39.514675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:48:03.277272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295257+0000","last_change":"2023-07-14T19:23:37.463231+0000","last_active":"2023-07-14T19:23:42.295257+0000","last_peered":"2023-07-14T19:23:42.295257+0000","last_clean":"2023-07-14T19:23:42.295257+0000","last_became_active":"2023-07-14T19:23:37.462571+0000","last_became_peered":"2023-07-14T19:23:37.462571+0000","last_unstale":"2023-07-14T19:23:42.295257+0000","last_undegraded":"2023-07-14T19:23:42.295257+0000","last_fullsized":"2023-07-14T19:23:42.295257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:07:42.060866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515814+0000","last_change":"2023-07-14T19:23:38.300276+0000","last_active":"2023-07-14T19:23:39.515814+0000","last_peered":"2023-07-14T19:23:39.515814+0000","last_clean":"2023-07-14T19:23:39.515814+0000","last_became_active":"2023-07-14T19:23:38.299984+0000","last_became_peered":"2023-07-14T19:23:38.299984+0000","last_unstale":"2023-07-14T19:23:39.515814+0000","last_undegraded":"2023-07-14T19:23:39.515814+0000","last_fullsized":"2023-07-14T19:23:39.515814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:19:10.316989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286669+0000","last_change":"2023-07-14T19:23:37.454592+0000","last_active":"2023-07-14T19:23:39.286669+0000","last_peered":"2023-07-14T19:23:39.286669+0000","last_clean":"2023-07-14T19:23:39.286669+0000","last_became_active":"2023-07-14T19:23:37.454266+0000","last_became_peered":"2023-07-14T19:23:37.454266+0000","last_unstale":"2023-07-14T19:23:39.286669+0000","last_undegraded":"2023-07-14T19:23:39.286669+0000","last_fullsized":"2023-07-14T19:23:39.286669+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:14.362148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.516093+0000","last_change":"2023-07-14T19:23:38.297945+0000","last_active":"2023-07-14T19:23:39.516093+0000","last_peered":"2023-07-14T19:23:39.516093+0000","last_clean":"2023-07-14T19:23:39.516093+0000","last_became_active":"2023-07-14T19:23:38.296733+0000","last_became_peered":"2023-07-14T19:23:38.296733+0000","last_unstale":"2023-07-14T19:23:39.516093+0000","last_undegraded":"2023-07-14T19:23:39.516093+0000","last_fullsized":"2023-07-14T19:23:39.516093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:38:24.778683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'5","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.288928+0000","last_change":"2023-07-14T19:23:37.460329+0000","last_active":"2023-07-14T19:23:42.288928+0000","last_peered":"2023-07-14T19:23:42.288928+0000","last_clean":"2023-07-14T19:23:42.288928+0000","last_became_active":"2023-07-14T19:23:37.460037+0000","last_became_peered":"2023-07-14T19:23:37.460037+0000","last_unstale":"2023-07-14T19:23:42.288928+0000","last_undegraded":"2023-07-14T19:23:42.288928+0000","last_fullsized":"2023-07-14T19:23:42.288928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:25:02.768120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286579+0000","last_change":"2023-07-14T19:23:38.293216+0000","last_active":"2023-07-14T19:23:39.286579+0000","last_peered":"2023-07-14T19:23:39.286579+0000","last_clean":"2023-07-14T19:23:39.286579+0000","last_became_active":"2023-07-14T19:23:38.292748+0000","last_became_peered":"2023-07-14T19:23:38.292748+0000","last_unstale":"2023-07-14T19:23:39.286579+0000","last_undegraded":"2023-07-14T19:23:39.286579+0000","last_fullsized":"2023-07-14T19:23:39.286579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:48.829793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295199+0000","last_change":"2023-07-14T19:23:37.462337+0000","last_active":"2023-07-14T19:23:42.295199+0000","last_peered":"2023-07-14T19:23:42.295199+0000","last_clean":"2023-07-14T19:23:42.295199+0000","last_became_active":"2023-07-14T19:23:37.462046+0000","last_became_peered":"2023-07-14T19:23:37.462046+0000","last_unstale":"2023-07-14T19:23:42.295199+0000","last_undegraded":"2023-07-14T19:23:42.295199+0000","last_fullsized":"2023-07-14T19:23:42.295199+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:07:29.186049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.516281+0000","last_change":"2023-07-14T19:23:38.297839+0000","last_active":"2023-07-14T19:23:39.516281+0000","last_peered":"2023-07-14T19:23:39.516281+0000","last_clean":"2023-07-14T19:23:39.516281+0000","last_became_active":"2023-07-14T19:23:38.296915+0000","last_became_peered":"2023-07-14T19:23:38.296915+0000","last_unstale":"2023-07-14T19:23:39.516281+0000","last_undegraded":"2023-07-14T19:23:39.516281+0000","last_fullsized":"2023-07-14T19:23:39.516281+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:44:44.107176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296502+0000","last_change":"2023-07-14T19:23:37.457637+0000","last_active":"2023-07-14T19:23:42.296502+0000","last_peered":"2023-07-14T19:23:42.296502+0000","last_clean":"2023-07-14T19:23:42.296502+0000","last_became_active":"2023-07-14T19:23:37.457305+0000","last_became_peered":"2023-07-14T19:23:37.457305+0000","last_unstale":"2023-07-14T19:23:42.296502+0000","last_undegraded":"2023-07-14T19:23:42.296502+0000","last_fullsized":"2023-07-14T19:23:42.296502+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:08:30.695027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283305+0000","last_change":"2023-07-14T19:23:38.295974+0000","last_active":"2023-07-14T19:23:39.283305+0000","last_peered":"2023-07-14T19:23:39.283305+0000","last_clean":"2023-07-14T19:23:39.283305+0000","last_became_active":"2023-07-14T19:23:38.295641+0000","last_became_peered":"2023-07-14T19:23:38.295641+0000","last_unstale":"2023-07-14T19:23:39.283305+0000","last_undegraded":"2023-07-14T19:23:39.283305+0000","last_fullsized":"2023-07-14T19:23:39.283305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:18:13.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515270+0000","last_change":"2023-07-14T19:23:37.457041+0000","last_active":"2023-07-14T19:23:39.515270+0000","last_peered":"2023-07-14T19:23:39.515270+0000","last_clean":"2023-07-14T19:23:39.515270+0000","last_became_active":"2023-07-14T19:23:37.456454+0000","last_became_peered":"2023-07-14T19:23:37.456454+0000","last_unstale":"2023-07-14T19:23:39.515270+0000","last_undegraded":"2023-07-14T19:23:39.515270+0000","last_fullsized":"2023-07-14T19:23:39.515270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:10:01.796045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515109+0000","last_change":"2023-07-14T19:23:38.293749+0000","last_active":"2023-07-14T19:23:39.515109+0000","last_peered":"2023-07-14T19:23:39.515109+0000","last_clean":"2023-07-14T19:23:39.515109+0000","last_became_active":"2023-07-14T19:23:38.293400+0000","last_became_peered":"2023-07-14T19:23:38.293400+0000","last_unstale":"2023-07-14T19:23:39.515109+0000","last_undegraded":"2023-07-14T19:23:39.515109+0000","last_fullsized":"2023-07-14T19:23:39.515109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:53.267258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'3","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:44.292441+0000","last_change":"2023-07-14T19:23:37.463783+0000","last_active":"2023-07-14T19:23:44.292441+0000","last_peered":"2023-07-14T19:23:44.292441+0000","last_clean":"2023-07-14T19:23:44.292441+0000","last_became_active":"2023-07-14T19:23:37.463421+0000","last_became_peered":"2023-07-14T19:23:37.463421+0000","last_unstale":"2023-07-14T19:23:44.292441+0000","last_undegraded":"2023-07-14T19:23:44.292441+0000","last_fullsized":"2023-07-14T19:23:44.292441+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:38.946539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515132+0000","last_change":"2023-07-14T19:23:38.300956+0000","last_active":"2023-07-14T19:23:39.515132+0000","last_peered":"2023-07-14T19:23:39.515132+0000","last_clean":"2023-07-14T19:23:39.515132+0000","last_became_active":"2023-07-14T19:23:38.300662+0000","last_became_peered":"2023-07-14T19:23:38.300662+0000","last_unstale":"2023-07-14T19:23:39.515132+0000","last_undegraded":"2023-07-14T19:23:39.515132+0000","last_fullsized":"2023-07-14T19:23:39.515132+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:59:21.546322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.476358+0000","last_change":"2023-07-14T19:23:37.458221+0000","last_active":"2023-07-14T19:23:41.476358+0000","last_peered":"2023-07-14T19:23:41.476358+0000","last_clean":"2023-07-14T19:23:41.476358+0000","last_became_active":"2023-07-14T19:23:37.457923+0000","last_became_peered":"2023-07-14T19:23:37.457923+0000","last_unstale":"2023-07-14T19:23:41.476358+0000","last_undegraded":"2023-07-14T19:23:41.476358+0000","last_fullsized":"2023-07-14T19:23:41.476358+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:51.733334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514538+0000","last_change":"2023-07-14T19:23:38.303977+0000","last_active":"2023-07-14T19:23:39.514538+0000","last_peered":"2023-07-14T19:23:39.514538+0000","last_clean":"2023-07-14T19:23:39.514538+0000","last_became_active":"2023-07-14T19:23:38.303333+0000","last_became_peered":"2023-07-14T19:23:38.303333+0000","last_unstale":"2023-07-14T19:23:39.514538+0000","last_undegraded":"2023-07-14T19:23:39.514538+0000","last_fullsized":"2023-07-14T19:23:39.514538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:28.958043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475704+0000","last_change":"2023-07-14T19:23:37.459181+0000","last_active":"2023-07-14T19:23:41.475704+0000","last_peered":"2023-07-14T19:23:41.475704+0000","last_clean":"2023-07-14T19:23:41.475704+0000","last_became_active":"2023-07-14T19:23:37.458890+0000","last_became_peered":"2023-07-14T19:23:37.458890+0000","last_unstale":"2023-07-14T19:23:41.475704+0000","last_undegraded":"2023-07-14T19:23:41.475704+0000","last_fullsized":"2023-07-14T19:23:41.475704+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:54.069559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515862+0000","last_change":"2023-07-14T19:23:38.295057+0000","last_active":"2023-07-14T19:23:39.515862+0000","last_peered":"2023-07-14T19:23:39.515862+0000","last_clean":"2023-07-14T19:23:39.515862+0000","last_became_active":"2023-07-14T19:23:38.294774+0000","last_became_peered":"2023-07-14T19:23:38.294774+0000","last_unstale":"2023-07-14T19:23:39.515862+0000","last_undegraded":"2023-07-14T19:23:39.515862+0000","last_fullsized":"2023-07-14T19:23:39.515862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:50:17.296593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'18","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:45.298945+0000","last_change":"2023-07-14T19:23:37.463196+0000","last_active":"2023-07-14T19:23:45.298945+0000","last_peered":"2023-07-14T19:23:45.298945+0000","last_clean":"2023-07-14T19:23:45.298945+0000","last_became_active":"2023-07-14T19:23:37.462867+0000","last_became_peered":"2023-07-14T19:23:37.462867+0000","last_unstale":"2023-07-14T19:23:45.298945+0000","last_undegraded":"2023-07-14T19:23:45.298945+0000","last_fullsized":"2023-07-14T19:23:45.298945+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:35.156443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36237,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":23,"num_write":19,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286612+0000","last_change":"2023-07-14T19:23:38.293249+0000","last_active":"2023-07-14T19:23:39.286612+0000","last_peered":"2023-07-14T19:23:39.286612+0000","last_clean":"2023-07-14T19:23:39.286612+0000","last_became_active":"2023-07-14T19:23:38.292938+0000","last_became_peered":"2023-07-14T19:23:38.292938+0000","last_unstale":"2023-07-14T19:23:39.286612+0000","last_undegraded":"2023-07-14T19:23:39.286612+0000","last_fullsized":"2023-07-14T19:23:39.286612+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:49:19.220723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"80'6","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296103+0000","last_change":"2023-07-14T19:23:37.460336+0000","last_active":"2023-07-14T19:23:42.296103+0000","last_peered":"2023-07-14T19:23:42.296103+0000","last_clean":"2023-07-14T19:23:42.296103+0000","last_became_active":"2023-07-14T19:23:37.460008+0000","last_became_peered":"2023-07-14T19:23:37.460008+0000","last_unstale":"2023-07-14T19:23:42.296103+0000","last_undegraded":"2023-07-14T19:23:42.296103+0000","last_fullsized":"2023-07-14T19:23:42.296103+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:31.610459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514712+0000","last_change":"2023-07-14T19:23:38.296819+0000","last_active":"2023-07-14T19:23:39.514712+0000","last_peered":"2023-07-14T19:23:39.514712+0000","last_clean":"2023-07-14T19:23:39.514712+0000","last_became_active":"2023-07-14T19:23:38.296225+0000","last_became_peered":"2023-07-14T19:23:38.296225+0000","last_unstale":"2023-07-14T19:23:39.514712+0000","last_undegraded":"2023-07-14T19:23:39.514712+0000","last_fullsized":"2023-07-14T19:23:39.514712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:43:30.925553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474332+0000","last_change":"2023-07-14T19:23:37.460018+0000","last_active":"2023-07-14T19:23:41.474332+0000","last_peered":"2023-07-14T19:23:41.474332+0000","last_clean":"2023-07-14T19:23:41.474332+0000","last_became_active":"2023-07-14T19:23:37.459713+0000","last_became_peered":"2023-07-14T19:23:37.459713+0000","last_unstale":"2023-07-14T19:23:41.474332+0000","last_undegraded":"2023-07-14T19:23:41.474332+0000","last_fullsized":"2023-07-14T19:23:41.474332+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:11.451517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515474+0000","last_change":"2023-07-14T19:23:38.301568+0000","last_active":"2023-07-14T19:23:39.515474+0000","last_peered":"2023-07-14T19:23:39.515474+0000","last_clean":"2023-07-14T19:23:39.515474+0000","last_became_active":"2023-07-14T19:23:38.301279+0000","last_became_peered":"2023-07-14T19:23:38.301279+0000","last_unstale":"2023-07-14T19:23:39.515474+0000","last_undegraded":"2023-07-14T19:23:39.515474+0000","last_fullsized":"2023-07-14T19:23:39.515474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:18.485184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514818+0000","last_change":"2023-07-14T19:23:37.460782+0000","last_active":"2023-07-14T19:23:39.514818+0000","last_peered":"2023-07-14T19:23:39.514818+0000","last_clean":"2023-07-14T19:23:39.514818+0000","last_became_active":"2023-07-14T19:23:37.460457+0000","last_became_peered":"2023-07-14T19:23:37.460457+0000","last_unstale":"2023-07-14T19:23:39.514818+0000","last_undegraded":"2023-07-14T19:23:39.514818+0000","last_fullsized":"2023-07-14T19:23:39.514818+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:19.994091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514572+0000","last_change":"2023-07-14T19:23:38.299681+0000","last_active":"2023-07-14T19:23:39.514572+0000","last_peered":"2023-07-14T19:23:39.514572+0000","last_clean":"2023-07-14T19:23:39.514572+0000","last_became_active":"2023-07-14T19:23:38.299345+0000","last_became_peered":"2023-07-14T19:23:38.299345+0000","last_unstale":"2023-07-14T19:23:39.514572+0000","last_undegraded":"2023-07-14T19:23:39.514572+0000","last_fullsized":"2023-07-14T19:23:39.514572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:58:44.871954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"80'15","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296518+0000","last_change":"2023-07-14T19:23:37.458413+0000","last_active":"2023-07-14T19:23:42.296518+0000","last_peered":"2023-07-14T19:23:42.296518+0000","last_clean":"2023-07-14T19:23:42.296518+0000","last_became_active":"2023-07-14T19:23:37.458108+0000","last_became_peered":"2023-07-14T19:23:37.458108+0000","last_unstale":"2023-07-14T19:23:42.296518+0000","last_undegraded":"2023-07-14T19:23:42.296518+0000","last_fullsized":"2023-07-14T19:23:42.296518+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:36.538174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515638+0000","last_change":"2023-07-14T19:23:38.296350+0000","last_active":"2023-07-14T19:23:39.515638+0000","last_peered":"2023-07-14T19:23:39.515638+0000","last_clean":"2023-07-14T19:23:39.515638+0000","last_became_active":"2023-07-14T19:23:38.296036+0000","last_became_peered":"2023-07-14T19:23:38.296036+0000","last_unstale":"2023-07-14T19:23:39.515638+0000","last_undegraded":"2023-07-14T19:23:39.515638+0000","last_fullsized":"2023-07-14T19:23:39.515638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:58:58.339059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.924053+0000","last_change":"2023-07-14T19:23:37.461647+0000","last_active":"2023-07-14T19:23:39.924053+0000","last_peered":"2023-07-14T19:23:39.924053+0000","last_clean":"2023-07-14T19:23:39.924053+0000","last_became_active":"2023-07-14T19:23:37.461325+0000","last_became_peered":"2023-07-14T19:23:37.461325+0000","last_unstale":"2023-07-14T19:23:39.924053+0000","last_undegraded":"2023-07-14T19:23:39.924053+0000","last_fullsized":"2023-07-14T19:23:39.924053+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:46.235600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472796+0000","last_change":"2023-07-14T19:23:38.300940+0000","last_active":"2023-07-14T19:23:39.472796+0000","last_peered":"2023-07-14T19:23:39.472796+0000","last_clean":"2023-07-14T19:23:39.472796+0000","last_became_active":"2023-07-14T19:23:38.300578+0000","last_became_peered":"2023-07-14T19:23:38.300578+0000","last_unstale":"2023-07-14T19:23:39.472796+0000","last_undegraded":"2023-07-14T19:23:39.472796+0000","last_fullsized":"2023-07-14T19:23:39.472796+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:50.864591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515474+0000","last_change":"2023-07-14T19:23:37.464251+0000","last_active":"2023-07-14T19:23:39.515474+0000","last_peered":"2023-07-14T19:23:39.515474+0000","last_clean":"2023-07-14T19:23:39.515474+0000","last_became_active":"2023-07-14T19:23:37.463793+0000","last_became_peered":"2023-07-14T19:23:37.463793+0000","last_unstale":"2023-07-14T19:23:39.515474+0000","last_undegraded":"2023-07-14T19:23:39.515474+0000","last_fullsized":"2023-07-14T19:23:39.515474+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:28:41.697534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:40.299231+0000","last_change":"2023-07-14T19:23:38.300077+0000","last_active":"2023-07-14T19:23:40.299231+0000","last_peered":"2023-07-14T19:23:40.299231+0000","last_clean":"2023-07-14T19:23:40.299231+0000","last_became_active":"2023-07-14T19:23:38.298708+0000","last_became_peered":"2023-07-14T19:23:38.298708+0000","last_unstale":"2023-07-14T19:23:40.299231+0000","last_undegraded":"2023-07-14T19:23:40.299231+0000","last_fullsized":"2023-07-14T19:23:40.299231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:13:31.564130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515610+0000","last_change":"2023-07-14T19:23:37.458361+0000","last_active":"2023-07-14T19:23:39.515610+0000","last_peered":"2023-07-14T19:23:39.515610+0000","last_clean":"2023-07-14T19:23:39.515610+0000","last_became_active":"2023-07-14T19:23:37.458108+0000","last_became_peered":"2023-07-14T19:23:37.458108+0000","last_unstale":"2023-07-14T19:23:39.515610+0000","last_undegraded":"2023-07-14T19:23:39.515610+0000","last_fullsized":"2023-07-14T19:23:39.515610+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:41:17.858983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284311+0000","last_change":"2023-07-14T19:23:38.301723+0000","last_active":"2023-07-14T19:23:39.284311+0000","last_peered":"2023-07-14T19:23:39.284311+0000","last_clean":"2023-07-14T19:23:39.284311+0000","last_became_active":"2023-07-14T19:23:38.301424+0000","last_became_peered":"2023-07-14T19:23:38.301424+0000","last_unstale":"2023-07-14T19:23:39.284311+0000","last_undegraded":"2023-07-14T19:23:39.284311+0000","last_fullsized":"2023-07-14T19:23:39.284311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:49.753010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515589+0000","last_change":"2023-07-14T19:23:38.296954+0000","last_active":"2023-07-14T19:23:39.515589+0000","last_peered":"2023-07-14T19:23:39.515589+0000","last_clean":"2023-07-14T19:23:39.515589+0000","last_became_active":"2023-07-14T19:23:38.296143+0000","last_became_peered":"2023-07-14T19:23:38.296143+0000","last_unstale":"2023-07-14T19:23:39.515589+0000","last_undegraded":"2023-07-14T19:23:39.515589+0000","last_fullsized":"2023-07-14T19:23:39.515589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:58:27.181508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296982+0000","last_change":"2023-07-14T19:23:37.460543+0000","last_active":"2023-07-14T19:23:42.296982+0000","last_peered":"2023-07-14T19:23:42.296982+0000","last_clean":"2023-07-14T19:23:42.296982+0000","last_became_active":"2023-07-14T19:23:37.460230+0000","last_became_peered":"2023-07-14T19:23:37.460230+0000","last_unstale":"2023-07-14T19:23:42.296982+0000","last_undegraded":"2023-07-14T19:23:42.296982+0000","last_fullsized":"2023-07-14T19:23:42.296982+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:17:14.842081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284020+0000","last_change":"2023-07-14T19:23:38.302414+0000","last_active":"2023-07-14T19:23:39.284020+0000","last_peered":"2023-07-14T19:23:39.284020+0000","last_clean":"2023-07-14T19:23:39.284020+0000","last_became_active":"2023-07-14T19:23:38.302089+0000","last_became_peered":"2023-07-14T19:23:38.302089+0000","last_unstale":"2023-07-14T19:23:39.284020+0000","last_undegraded":"2023-07-14T19:23:39.284020+0000","last_fullsized":"2023-07-14T19:23:39.284020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:09:15.082408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515110+0000","last_change":"2023-07-14T19:23:37.462245+0000","last_active":"2023-07-14T19:23:39.515110+0000","last_peered":"2023-07-14T19:23:39.515110+0000","last_clean":"2023-07-14T19:23:39.515110+0000","last_became_active":"2023-07-14T19:23:37.461911+0000","last_became_peered":"2023-07-14T19:23:37.461911+0000","last_unstale":"2023-07-14T19:23:39.515110+0000","last_undegraded":"2023-07-14T19:23:39.515110+0000","last_fullsized":"2023-07-14T19:23:39.515110+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:53.399246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":337,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515010+0000","last_change":"2023-07-14T19:22:39.254277+0000","last_active":"2023-07-14T19:23:39.515010+0000","last_peered":"2023-07-14T19:23:39.515010+0000","last_clean":"2023-07-14T19:23:39.515010+0000","last_became_active":"2023-07-14T19:22:39.253670+0000","last_became_peered":"2023-07-14T19:22:39.253670+0000","last_unstale":"2023-07-14T19:23:39.515010+0000","last_undegraded":"2023-07-14T19:23:39.515010+0000","last_fullsized":"2023-07-14T19:23:39.515010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:19:52.229492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:19:52.229492+0000","last_clean_scrub_stamp":"2023-07-14T19:19:52.229492+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:22.025952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514166+0000","last_change":"2023-07-14T19:23:38.296671+0000","last_active":"2023-07-14T19:23:39.514166+0000","last_peered":"2023-07-14T19:23:39.514166+0000","last_clean":"2023-07-14T19:23:39.514166+0000","last_became_active":"2023-07-14T19:23:38.296175+0000","last_became_peered":"2023-07-14T19:23:38.296175+0000","last_unstale":"2023-07-14T19:23:39.514166+0000","last_undegraded":"2023-07-14T19:23:39.514166+0000","last_fullsized":"2023-07-14T19:23:39.514166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:35:01.140553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295431+0000","last_change":"2023-07-14T19:23:37.459269+0000","last_active":"2023-07-14T19:23:42.295431+0000","last_peered":"2023-07-14T19:23:42.295431+0000","last_clean":"2023-07-14T19:23:42.295431+0000","last_became_active":"2023-07-14T19:23:37.458999+0000","last_became_peered":"2023-07-14T19:23:37.458999+0000","last_unstale":"2023-07-14T19:23:42.295431+0000","last_undegraded":"2023-07-14T19:23:42.295431+0000","last_fullsized":"2023-07-14T19:23:42.295431+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:45.895670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"80'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475684+0000","last_change":"2023-07-14T19:23:37.463425+0000","last_active":"2023-07-14T19:23:41.475684+0000","last_peered":"2023-07-14T19:23:41.475684+0000","last_clean":"2023-07-14T19:23:41.475684+0000","last_became_active":"2023-07-14T19:23:37.463005+0000","last_became_peered":"2023-07-14T19:23:37.463005+0000","last_unstale":"2023-07-14T19:23:41.475684+0000","last_undegraded":"2023-07-14T19:23:41.475684+0000","last_fullsized":"2023-07-14T19:23:41.475684+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:22:15.482538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515091+0000","last_change":"2023-07-14T19:23:38.295556+0000","last_active":"2023-07-14T19:23:39.515091+0000","last_peered":"2023-07-14T19:23:39.515091+0000","last_clean":"2023-07-14T19:23:39.515091+0000","last_became_active":"2023-07-14T19:23:38.295233+0000","last_became_peered":"2023-07-14T19:23:38.295233+0000","last_unstale":"2023-07-14T19:23:39.515091+0000","last_undegraded":"2023-07-14T19:23:39.515091+0000","last_fullsized":"2023-07-14T19:23:39.515091+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:29.024672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515774+0000","last_change":"2023-07-14T19:23:37.464752+0000","last_active":"2023-07-14T19:23:39.515774+0000","last_peered":"2023-07-14T19:23:39.515774+0000","last_clean":"2023-07-14T19:23:39.515774+0000","last_became_active":"2023-07-14T19:23:37.464451+0000","last_became_peered":"2023-07-14T19:23:37.464451+0000","last_unstale":"2023-07-14T19:23:39.515774+0000","last_undegraded":"2023-07-14T19:23:39.515774+0000","last_fullsized":"2023-07-14T19:23:39.515774+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:30:13.403469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284070+0000","last_change":"2023-07-14T19:23:38.298475+0000","last_active":"2023-07-14T19:23:39.284070+0000","last_peered":"2023-07-14T19:23:39.284070+0000","last_clean":"2023-07-14T19:23:39.284070+0000","last_became_active":"2023-07-14T19:23:38.298141+0000","last_became_peered":"2023-07-14T19:23:38.298141+0000","last_unstale":"2023-07-14T19:23:39.284070+0000","last_undegraded":"2023-07-14T19:23:39.284070+0000","last_fullsized":"2023-07-14T19:23:39.284070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:02.834103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514928+0000","last_change":"2023-07-14T19:23:37.457182+0000","last_active":"2023-07-14T19:23:39.514928+0000","last_peered":"2023-07-14T19:23:39.514928+0000","last_clean":"2023-07-14T19:23:39.514928+0000","last_became_active":"2023-07-14T19:23:37.456825+0000","last_became_peered":"2023-07-14T19:23:37.456825+0000","last_unstale":"2023-07-14T19:23:39.514928+0000","last_undegraded":"2023-07-14T19:23:39.514928+0000","last_fullsized":"2023-07-14T19:23:39.514928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:03.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514703+0000","last_change":"2023-07-14T19:23:38.303732+0000","last_active":"2023-07-14T19:23:39.514703+0000","last_peered":"2023-07-14T19:23:39.514703+0000","last_clean":"2023-07-14T19:23:39.514703+0000","last_became_active":"2023-07-14T19:23:38.303296+0000","last_became_peered":"2023-07-14T19:23:38.303296+0000","last_unstale":"2023-07-14T19:23:39.514703+0000","last_undegraded":"2023-07-14T19:23:39.514703+0000","last_fullsized":"2023-07-14T19:23:39.514703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:55:39.991626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475191+0000","last_change":"2023-07-14T19:23:37.461183+0000","last_active":"2023-07-14T19:23:41.475191+0000","last_peered":"2023-07-14T19:23:41.475191+0000","last_clean":"2023-07-14T19:23:41.475191+0000","last_became_active":"2023-07-14T19:23:37.460876+0000","last_became_peered":"2023-07-14T19:23:37.460876+0000","last_unstale":"2023-07-14T19:23:41.475191+0000","last_undegraded":"2023-07-14T19:23:41.475191+0000","last_fullsized":"2023-07-14T19:23:41.475191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:31.746901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286421+0000","last_change":"2023-07-14T19:23:38.292032+0000","last_active":"2023-07-14T19:23:39.286421+0000","last_peered":"2023-07-14T19:23:39.286421+0000","last_clean":"2023-07-14T19:23:39.286421+0000","last_became_active":"2023-07-14T19:23:38.291721+0000","last_became_peered":"2023-07-14T19:23:38.291721+0000","last_unstale":"2023-07-14T19:23:39.286421+0000","last_undegraded":"2023-07-14T19:23:39.286421+0000","last_fullsized":"2023-07-14T19:23:39.286421+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:58.334225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283442+0000","last_change":"2023-07-14T19:23:37.465597+0000","last_active":"2023-07-14T19:23:39.283442+0000","last_peered":"2023-07-14T19:23:39.283442+0000","last_clean":"2023-07-14T19:23:39.283442+0000","last_became_active":"2023-07-14T19:23:37.465233+0000","last_became_peered":"2023-07-14T19:23:37.465233+0000","last_unstale":"2023-07-14T19:23:39.283442+0000","last_undegraded":"2023-07-14T19:23:39.283442+0000","last_fullsized":"2023-07-14T19:23:39.283442+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:55.848783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472581+0000","last_change":"2023-07-14T19:23:38.295185+0000","last_active":"2023-07-14T19:23:39.472581+0000","last_peered":"2023-07-14T19:23:39.472581+0000","last_clean":"2023-07-14T19:23:39.472581+0000","last_became_active":"2023-07-14T19:23:38.294855+0000","last_became_peered":"2023-07-14T19:23:38.294855+0000","last_unstale":"2023-07-14T19:23:39.472581+0000","last_undegraded":"2023-07-14T19:23:39.472581+0000","last_fullsized":"2023-07-14T19:23:39.472581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:48.045146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:40.262742+0000","last_change":"2023-07-14T19:23:37.463517+0000","last_active":"2023-07-14T19:23:40.262742+0000","last_peered":"2023-07-14T19:23:40.262742+0000","last_clean":"2023-07-14T19:23:40.262742+0000","last_became_active":"2023-07-14T19:23:37.463158+0000","last_became_peered":"2023-07-14T19:23:37.463158+0000","last_unstale":"2023-07-14T19:23:40.262742+0000","last_undegraded":"2023-07-14T19:23:40.262742+0000","last_fullsized":"2023-07-14T19:23:40.262742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:01.142000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283721+0000","last_change":"2023-07-14T19:23:38.305375+0000","last_active":"2023-07-14T19:23:39.283721+0000","last_peered":"2023-07-14T19:23:39.283721+0000","last_clean":"2023-07-14T19:23:39.283721+0000","last_became_active":"2023-07-14T19:23:38.305084+0000","last_became_peered":"2023-07-14T19:23:38.305084+0000","last_unstale":"2023-07-14T19:23:39.283721+0000","last_undegraded":"2023-07-14T19:23:39.283721+0000","last_fullsized":"2023-07-14T19:23:39.283721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:08:51.652651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474944+0000","last_change":"2023-07-14T19:23:37.458672+0000","last_active":"2023-07-14T19:23:41.474944+0000","last_peered":"2023-07-14T19:23:41.474944+0000","last_clean":"2023-07-14T19:23:41.474944+0000","last_became_active":"2023-07-14T19:23:37.458359+0000","last_became_peered":"2023-07-14T19:23:37.458359+0000","last_unstale":"2023-07-14T19:23:41.474944+0000","last_undegraded":"2023-07-14T19:23:41.474944+0000","last_fullsized":"2023-07-14T19:23:41.474944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:00:25.922910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515725+0000","last_change":"2023-07-14T19:23:38.301741+0000","last_active":"2023-07-14T19:23:39.515725+0000","last_peered":"2023-07-14T19:23:39.515725+0000","last_clean":"2023-07-14T19:23:39.515725+0000","last_became_active":"2023-07-14T19:23:38.301430+0000","last_became_peered":"2023-07-14T19:23:38.301430+0000","last_unstale":"2023-07-14T19:23:39.515725+0000","last_undegraded":"2023-07-14T19:23:39.515725+0000","last_fullsized":"2023-07-14T19:23:39.515725+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:13.088226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296558+0000","last_change":"2023-07-14T19:23:37.461764+0000","last_active":"2023-07-14T19:23:42.296558+0000","last_peered":"2023-07-14T19:23:42.296558+0000","last_clean":"2023-07-14T19:23:42.296558+0000","last_became_active":"2023-07-14T19:23:37.460910+0000","last_became_peered":"2023-07-14T19:23:37.460910+0000","last_unstale":"2023-07-14T19:23:42.296558+0000","last_undegraded":"2023-07-14T19:23:42.296558+0000","last_fullsized":"2023-07-14T19:23:42.296558+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:39.478046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515703+0000","last_change":"2023-07-14T19:23:38.296214+0000","last_active":"2023-07-14T19:23:39.515703+0000","last_peered":"2023-07-14T19:23:39.515703+0000","last_clean":"2023-07-14T19:23:39.515703+0000","last_became_active":"2023-07-14T19:23:38.295148+0000","last_became_peered":"2023-07-14T19:23:38.295148+0000","last_unstale":"2023-07-14T19:23:39.515703+0000","last_undegraded":"2023-07-14T19:23:39.515703+0000","last_fullsized":"2023-07-14T19:23:39.515703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:30.117202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515939+0000","last_change":"2023-07-14T19:23:37.458245+0000","last_active":"2023-07-14T19:23:39.515939+0000","last_peered":"2023-07-14T19:23:39.515939+0000","last_clean":"2023-07-14T19:23:39.515939+0000","last_became_active":"2023-07-14T19:23:37.457784+0000","last_became_peered":"2023-07-14T19:23:37.457784+0000","last_unstale":"2023-07-14T19:23:39.515939+0000","last_undegraded":"2023-07-14T19:23:39.515939+0000","last_fullsized":"2023-07-14T19:23:39.515939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:47:00.389865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515321+0000","last_change":"2023-07-14T19:23:38.299823+0000","last_active":"2023-07-14T19:23:39.515321+0000","last_peered":"2023-07-14T19:23:39.515321+0000","last_clean":"2023-07-14T19:23:39.515321+0000","last_became_active":"2023-07-14T19:23:38.299496+0000","last_became_peered":"2023-07-14T19:23:38.299496+0000","last_unstale":"2023-07-14T19:23:39.515321+0000","last_undegraded":"2023-07-14T19:23:39.515321+0000","last_fullsized":"2023-07-14T19:23:39.515321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:23:40.353266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514282+0000","last_change":"2023-07-14T19:23:37.463439+0000","last_active":"2023-07-14T19:23:39.514282+0000","last_peered":"2023-07-14T19:23:39.514282+0000","last_clean":"2023-07-14T19:23:39.514282+0000","last_became_active":"2023-07-14T19:23:37.463100+0000","last_became_peered":"2023-07-14T19:23:37.463100+0000","last_unstale":"2023-07-14T19:23:39.514282+0000","last_undegraded":"2023-07-14T19:23:39.514282+0000","last_fullsized":"2023-07-14T19:23:39.514282+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:56.140535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515366+0000","last_change":"2023-07-14T19:23:38.296265+0000","last_active":"2023-07-14T19:23:39.515366+0000","last_peered":"2023-07-14T19:23:39.515366+0000","last_clean":"2023-07-14T19:23:39.515366+0000","last_became_active":"2023-07-14T19:23:38.295918+0000","last_became_peered":"2023-07-14T19:23:38.295918+0000","last_unstale":"2023-07-14T19:23:39.515366+0000","last_undegraded":"2023-07-14T19:23:39.515366+0000","last_fullsized":"2023-07-14T19:23:39.515366+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:38.769993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472522+0000","last_change":"2023-07-14T19:23:38.300200+0000","last_active":"2023-07-14T19:23:39.472522+0000","last_peered":"2023-07-14T19:23:39.472522+0000","last_clean":"2023-07-14T19:23:39.472522+0000","last_became_active":"2023-07-14T19:23:38.299979+0000","last_became_peered":"2023-07-14T19:23:38.299979+0000","last_unstale":"2023-07-14T19:23:39.472522+0000","last_undegraded":"2023-07-14T19:23:39.472522+0000","last_fullsized":"2023-07-14T19:23:39.472522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:00.789219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514842+0000","last_change":"2023-07-14T19:23:38.300696+0000","last_active":"2023-07-14T19:23:39.514842+0000","last_peered":"2023-07-14T19:23:39.514842+0000","last_clean":"2023-07-14T19:23:39.514842+0000","last_became_active":"2023-07-14T19:23:38.300377+0000","last_became_peered":"2023-07-14T19:23:38.300377+0000","last_unstale":"2023-07-14T19:23:39.514842+0000","last_undegraded":"2023-07-14T19:23:39.514842+0000","last_fullsized":"2023-07-14T19:23:39.514842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:39:05.440224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514814+0000","last_change":"2023-07-14T19:23:38.298616+0000","last_active":"2023-07-14T19:23:39.514814+0000","last_peered":"2023-07-14T19:23:39.514814+0000","last_clean":"2023-07-14T19:23:39.514814+0000","last_became_active":"2023-07-14T19:23:38.298313+0000","last_became_peered":"2023-07-14T19:23:38.298313+0000","last_unstale":"2023-07-14T19:23:39.514814+0000","last_undegraded":"2023-07-14T19:23:39.514814+0000","last_fullsized":"2023-07-14T19:23:39.514814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:46:49.950796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514982+0000","last_change":"2023-07-14T19:23:38.297185+0000","last_active":"2023-07-14T19:23:39.514982+0000","last_peered":"2023-07-14T19:23:39.514982+0000","last_clean":"2023-07-14T19:23:39.514982+0000","last_became_active":"2023-07-14T19:23:38.296883+0000","last_became_peered":"2023-07-14T19:23:38.296883+0000","last_unstale":"2023-07-14T19:23:39.514982+0000","last_undegraded":"2023-07-14T19:23:39.514982+0000","last_fullsized":"2023-07-14T19:23:39.514982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:08:27.715078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514907+0000","last_change":"2023-07-14T19:23:38.296636+0000","last_active":"2023-07-14T19:23:39.514907+0000","last_peered":"2023-07-14T19:23:39.514907+0000","last_clean":"2023-07-14T19:23:39.514907+0000","last_became_active":"2023-07-14T19:23:38.296289+0000","last_became_peered":"2023-07-14T19:23:38.296289+0000","last_unstale":"2023-07-14T19:23:39.514907+0000","last_undegraded":"2023-07-14T19:23:39.514907+0000","last_fullsized":"2023-07-14T19:23:39.514907+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:18:30.937689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472626+0000","last_change":"2023-07-14T19:23:38.300340+0000","last_active":"2023-07-14T19:23:39.472626+0000","last_peered":"2023-07-14T19:23:39.472626+0000","last_clean":"2023-07-14T19:23:39.472626+0000","last_became_active":"2023-07-14T19:23:38.299520+0000","last_became_peered":"2023-07-14T19:23:38.299520+0000","last_unstale":"2023-07-14T19:23:39.472626+0000","last_undegraded":"2023-07-14T19:23:39.472626+0000","last_fullsized":"2023-07-14T19:23:39.472626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:03:47.657130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515538+0000","last_change":"2023-07-14T19:23:38.294136+0000","last_active":"2023-07-14T19:23:39.515538+0000","last_peered":"2023-07-14T19:23:39.515538+0000","last_clean":"2023-07-14T19:23:39.515538+0000","last_became_active":"2023-07-14T19:23:38.293785+0000","last_became_peered":"2023-07-14T19:23:38.293785+0000","last_unstale":"2023-07-14T19:23:39.515538+0000","last_undegraded":"2023-07-14T19:23:39.515538+0000","last_fullsized":"2023-07-14T19:23:39.515538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:21.794486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284002+0000","last_change":"2023-07-14T19:23:38.305648+0000","last_active":"2023-07-14T19:23:39.284002+0000","last_peered":"2023-07-14T19:23:39.284002+0000","last_clean":"2023-07-14T19:23:39.284002+0000","last_became_active":"2023-07-14T19:23:38.305314+0000","last_became_peered":"2023-07-14T19:23:38.305314+0000","last_unstale":"2023-07-14T19:23:39.284002+0000","last_undegraded":"2023-07-14T19:23:39.284002+0000","last_fullsized":"2023-07-14T19:23:39.284002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:01:05.266523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515131+0000","last_change":"2023-07-14T19:23:38.298189+0000","last_active":"2023-07-14T19:23:39.515131+0000","last_peered":"2023-07-14T19:23:39.515131+0000","last_clean":"2023-07-14T19:23:39.515131+0000","last_became_active":"2023-07-14T19:23:38.297781+0000","last_became_peered":"2023-07-14T19:23:38.297781+0000","last_unstale":"2023-07-14T19:23:39.515131+0000","last_undegraded":"2023-07-14T19:23:39.515131+0000","last_fullsized":"2023-07-14T19:23:39.515131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:49:59.706932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283450+0000","last_change":"2023-07-14T19:23:38.293567+0000","last_active":"2023-07-14T19:23:39.283450+0000","last_peered":"2023-07-14T19:23:39.283450+0000","last_clean":"2023-07-14T19:23:39.283450+0000","last_became_active":"2023-07-14T19:23:38.293222+0000","last_became_peered":"2023-07-14T19:23:38.293222+0000","last_unstale":"2023-07-14T19:23:39.283450+0000","last_undegraded":"2023-07-14T19:23:39.283450+0000","last_fullsized":"2023-07-14T19:23:39.283450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:28.545521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515396+0000","last_change":"2023-07-14T19:23:38.294168+0000","last_active":"2023-07-14T19:23:39.515396+0000","last_peered":"2023-07-14T19:23:39.515396+0000","last_clean":"2023-07-14T19:23:39.515396+0000","last_became_active":"2023-07-14T19:23:38.293665+0000","last_became_peered":"2023-07-14T19:23:38.293665+0000","last_unstale":"2023-07-14T19:23:39.515396+0000","last_undegraded":"2023-07-14T19:23:39.515396+0000","last_fullsized":"2023-07-14T19:23:39.515396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:49.466175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515479+0000","last_change":"2023-07-14T19:23:38.296535+0000","last_active":"2023-07-14T19:23:39.515479+0000","last_peered":"2023-07-14T19:23:39.515479+0000","last_clean":"2023-07-14T19:23:39.515479+0000","last_became_active":"2023-07-14T19:23:38.296189+0000","last_became_peered":"2023-07-14T19:23:38.296189+0000","last_unstale":"2023-07-14T19:23:39.515479+0000","last_undegraded":"2023-07-14T19:23:39.515479+0000","last_fullsized":"2023-07-14T19:23:39.515479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:36:54.183312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472559+0000","last_change":"2023-07-14T19:23:38.296867+0000","last_active":"2023-07-14T19:23:39.472559+0000","last_peered":"2023-07-14T19:23:39.472559+0000","last_clean":"2023-07-14T19:23:39.472559+0000","last_became_active":"2023-07-14T19:23:38.296560+0000","last_became_peered":"2023-07-14T19:23:38.296560+0000","last_unstale":"2023-07-14T19:23:39.472559+0000","last_undegraded":"2023-07-14T19:23:39.472559+0000","last_fullsized":"2023-07-14T19:23:39.472559+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:48:16.893808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":39087,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":28,"num_read_kb":39,"num_write":111,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":100763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98,"ondisk_log_size":98,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547212,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":860160,"data_stored":413577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":10,"up_from":68,"seq":292057776143,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28512,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716640,"statfs":{"total":95995035648,"available":95965839360,"internally_reserved":0,"allocated":1531904,"data_stored":1041504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40799999999999997}]},{"osd":2,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.61399999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.53800000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.498},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.55300000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.65900000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.59299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.72599999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.68200000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":9,"up_from":63,"seq":270582939667,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":414159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.307},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.6080000000000001,"5min":2.6080000000000001,"15min":2.6080000000000001},"last":0.41799999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.657,"5min":2.657,"15min":2.657},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":0.52400000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":2.7589999999999999,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.497},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.59399999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.6699999999999999,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.9740000000000002,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.53500000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.69099999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.7309999999999999,"5min":2.7309999999999999,"15min":2.7309999999999999},"last":0.66500000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":0.70899999999999996}]}]},{"osd":8,"up_from":56,"seq":240518168600,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":413984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.443},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.72599999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.47499999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.53600000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.75}]},{"osd":5,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.65700000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.51900000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.628},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76500000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:23:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.56200000000000006}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.67400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":7,"up_from":48,"seq":206158430235,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1466368,"data_stored":1005101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73099999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.82299999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.43099999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42499999999999999,"15min":0.42099999999999999},"min":{"1min":0.32900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78600000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.45800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.83699999999999997}]},{"osd":4,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.502,"15min":0.49399999999999999},"min":{"1min":0.36299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.69399999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.621,"15min":0.624},"min":{"1min":0.47699999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66600000000000004,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.76100000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.51300000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66400000000000003},"min":{"1min":0.41199999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.77500000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.753}]},{"osd":9,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.90900000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.81599999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":6,"up_from":43,"seq":184683593761,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":413667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.30299999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60999999999999999,"5min":0.62,"15min":0.62},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.72799999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.35399999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.54000000000000004,"15min":0.53600000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.39500000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.61699999999999999,"15min":0.60899999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.62},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.63700000000000001,"15min":0.621},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52000000000000002,"15min":0.51300000000000001},"min":{"1min":0.30599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.54300000000000004,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.875,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.40999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.66300000000000003,"15min":0.64700000000000002},"min":{"1min":0.45100000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.60499999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.67200000000000004,"15min":0.66000000000000003},"min":{"1min":0.46700000000000003,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61499999999999999,"15min":0.61199999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64100000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.57099999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.71399999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.44600000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":432233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53200000000000003,"15min":0.52200000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83999999999999997,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.46600000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53000000000000003,"15min":0.51700000000000002},"min":{"1min":0.42199999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.85899999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.54700000000000004,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.878,"5min":1.028,"15min":1.028},"last":0.68000000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.40300000000000002},"min":{"1min":0.27600000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.433,"15min":0.42499999999999999},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.58099999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.873},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.89400000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.61699999999999999,"15min":0.60599999999999998},"min":{"1min":0.503,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65900000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.73099999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.95599999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:23:15 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.97899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":872448,"data_stored":414271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.52500000000000002,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.84599999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.51400000000000001,"15min":0.46800000000000003},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.51800000000000002,"15min":0.5},"min":{"1min":0.36599999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.54200000000000004,"15min":0.52300000000000002},"min":{"1min":0.44400000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48799999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.432,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.371},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.433},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.42699999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59099999999999997,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.58699999999999997,"15min":0.58199999999999996},"min":{"1min":0.41199999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61499999999999999,"15min":0.60599999999999998},"min":{"1min":0.42299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.623}]},{"osd":6,"last update":"Fri Jul 14 19:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.46999999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67600000000000005,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.434,"15min":0.434},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.73099999999999998,"15min":0.72699999999999998},"min":{"1min":0.55900000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.58599999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.56299999999999994}]},{"osd":9,"last update":"Fri Jul 14 19:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.60499999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.67800000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":434198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.35299999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.499,"15min":0.48999999999999999},"min":{"1min":0.35299999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52200000000000002,"15min":0.51800000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.52400000000000002,"15min":0.5},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.56399999999999995,"15min":0.55000000000000004},"min":{"1min":0.45800000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.39200000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.50700000000000001},"min":{"1min":0.438,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72599999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69299999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.69699999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.44,"15min":0.42799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.624},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.46899999999999997,"15min":0.46200000000000002},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.80300000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.61899999999999999,"15min":0.61099999999999999},"min":{"1min":0.47599999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.434,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82799999999999996,"5min":3.6579999999999999,"15min":3.6579999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.79900000000000004,"15min":0.82899999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.79500000000000004,"5min":3.8769999999999998,"15min":3.8769999999999998},"last":0.45900000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.82999999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.78800000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.75},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.879}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":3,"up_from":24,"seq":103079215148,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":868352,"data_stored":413681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.34899999999999998},"min":{"1min":0.30399999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.432},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.39000000000000001,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.33900000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.54200000000000004,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.436,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.61899999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.48999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.57899999999999996,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.72499999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.57699999999999996,"15min":0.55500000000000005},"min":{"1min":0.47299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.625,"15min":0.59999999999999998},"min":{"1min":0.41499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.95799999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.51100000000000001,"5min":0.37,"15min":0.37},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.54900000000000004,"15min":0.53500000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.752}]},{"osd":6,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.78200000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.443,"5min":0.436,"15min":0.436},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.87},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.73099999999999998,"15min":0.71699999999999997},"min":{"1min":0.54300000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":1.0009999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.54800000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.85199999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.92100000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986215,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":864256,"data_stored":413599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.34499999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.44,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73999999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.46200000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.36799999999999999,"15min":0.33800000000000002},"min":{"1min":0.29499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.78200000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.39700000000000002,"15min":0.377},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.81599999999999995,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.54500000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.755},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.51900000000000002},"min":{"1min":0.48799999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88900000000000001,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.40000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84799999999999998,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.629},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.47099999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.48599999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.59199999999999997,"15min":0.57199999999999995},"min":{"1min":0.53100000000000003,"5min":0.372,"15min":0.372},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.85199999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.57199999999999995,"15min":0.56899999999999995},"min":{"1min":0.42299999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.94799999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.53900000000000003,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.496},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.442,"5min":0.377,"15min":0.377},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.70999999999999996}]},{"osd":8,"last update":"Fri Jul 14 19:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69099999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.77900000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.72699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":5,"up_from":36,"seq":154618822692,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1478656,"data_stored":1005042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.40500000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54300000000000004,"15min":0.53500000000000003},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55100000000000005}]},{"osd":1,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.755,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.372,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.29099999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.39300000000000002},"min":{"1min":0.29099999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.61699999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.46400000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.57099999999999995,"15min":0.56299999999999994},"min":{"1min":0.42499999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.72699999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54200000000000004,"15min":0.53900000000000003},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.39300000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.65600000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.79200000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.51300000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64600000000000002,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.82399999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.86199999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:23:16 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.84799999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.89700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":18929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":37929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":37767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:23:46.457 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-14T19:23:46.458 INFO:tasks.cephadm:Setting up client nodes... 2023-07-14T19:23:46.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-14T19:23:46.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:46 smithi053 ceph-mon[96565]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:46 smithi053 ceph-mon[96565]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:46 smithi053 ceph-mon[96565]: pgmap v252: 97 pgs: 97 active+clean; 615 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8.3 KiB/s wr, 17 op/s 2023-07-14T19:23:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:46 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3192641406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:46 smithi053 ceph-mon[96565]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:23:46.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:46 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1070486337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:46 smithi070 ceph-mon[98015]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:46 smithi070 ceph-mon[98015]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:46 smithi070 ceph-mon[98015]: pgmap v252: 97 pgs: 97 active+clean; 615 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8.3 KiB/s wr, 17 op/s 2023-07-14T19:23:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:46 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3192641406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:46.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:46 smithi070 ceph-mon[98015]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:23:46.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:46 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1070486337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:46.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:46 smithi104 ceph-mon[98435]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:46.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:46 smithi104 ceph-mon[98435]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:46.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:46 smithi104 ceph-mon[98435]: pgmap v252: 97 pgs: 97 active+clean; 615 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8.3 KiB/s wr, 17 op/s 2023-07-14T19:23:46.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:46 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3192641406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:23:46.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:46 smithi104 ceph-mon[98435]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:23:46.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:46 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1070486337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:47.046 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:47.566 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:47 smithi053 ceph-mon[96565]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:23:47.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:47 smithi070 ceph-mon[98015]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:23:47.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:47 smithi104 ceph-mon[98435]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:23:48.466 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 -- 172.21.15.53:0/2961921955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 msgr2=0x7f96ac102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:48.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 --2- 172.21.15.53:0/2961921955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 0x7f96ac102b60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f96a8009990 tx=0x7f96a8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 -- 172.21.15.53:0/2961921955 shutdown_connections 2023-07-14T19:23:48.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 --2- 172.21.15.53:0/2961921955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96ac1059c0 0x7f96ac107e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 --2- 172.21.15.53:0/2961921955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 0x7f96ac102b60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 --2- 172.21.15.53:0/2961921955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac1030a0 0x7f96ac105480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 -- 172.21.15.53:0/2961921955 >> 172.21.15.53:0/2961921955 conn(0x7f96ac0fa240 msgr2=0x7f96ac0fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:48.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 -- 172.21.15.53:0/2961921955 shutdown_connections 2023-07-14T19:23:48.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 -- 172.21.15.53:0/2961921955 wait complete. 2023-07-14T19:23:48.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 Processor -- start 2023-07-14T19:23:48.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.464+0000 7f96b3279700 1 -- start start 2023-07-14T19:23:48.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b3279700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 0x7f96ac1959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:48.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b3279700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac1030a0 0x7f96ac195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:48.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b3279700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96ac1059c0 0x7f96ac19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:48.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b3279700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac19d580 con 0x7f96ac100780 2023-07-14T19:23:48.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b3279700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac19d6c0 con 0x7f96ac1059c0 2023-07-14T19:23:48.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b3279700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac19d850 con 0x7f96ac1030a0 2023-07-14T19:23:48.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1015700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 0x7f96ac1959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:48.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1015700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 0x7f96ac1959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50420/0 (socket says 172.21.15.53:50420) 2023-07-14T19:23:48.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1015700 1 -- 172.21.15.53:0/3844506567 learned_addr learned my addr 172.21.15.53:0/3844506567 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:48.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1816700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96ac1059c0 0x7f96ac19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:48.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b0814700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac1030a0 0x7f96ac195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:48.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1015700 1 -- 172.21.15.53:0/3844506567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac1030a0 msgr2=0x7f96ac195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:48.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1015700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac1030a0 0x7f96ac195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1015700 1 -- 172.21.15.53:0/3844506567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96ac1059c0 msgr2=0x7f96ac19cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:48.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1015700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96ac1059c0 0x7f96ac19cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1015700 1 -- 172.21.15.53:0/3844506567 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a80095f0 con 0x7f96ac100780 2023-07-14T19:23:48.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b1015700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 0x7f96ac1959f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f96a8005910 tx=0x7f96a800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:48.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96a27fc700 1 -- 172.21.15.53:0/3844506567 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a801d070 con 0x7f96ac100780 2023-07-14T19:23:48.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96a27fc700 1 -- 172.21.15.53:0/3844506567 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96a8022470 con 0x7f96ac100780 2023-07-14T19:23:48.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac19dad0 con 0x7f96ac100780 2023-07-14T19:23:48.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96a27fc700 1 -- 172.21.15.53:0/3844506567 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8017650 con 0x7f96ac100780 2023-07-14T19:23:48.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.465+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac19df70 con 0x7f96ac100780 2023-07-14T19:23:48.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.466+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac0fbdb0 con 0x7f96ac100780 2023-07-14T19:23:48.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.467+0000 7f96a27fc700 1 -- 172.21.15.53:0/3844506567 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f96a80225e0 con 0x7f96ac100780 2023-07-14T19:23:48.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.468+0000 7f96a27fc700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96980a11e0 0x7f96980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:48.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.468+0000 7f96b0814700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96980a11e0 0x7f96980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:48.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.468+0000 7f96a27fc700 1 -- 172.21.15.53:0/3844506567 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f96a80cdee0 con 0x7f96ac100780 2023-07-14T19:23:48.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.469+0000 7f96b0814700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96980a11e0 0x7f96980a3690 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f969c005950 tx=0x7f969c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:48.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.471+0000 7f96a27fc700 1 -- 172.21.15.53:0/3844506567 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f96a8097710 con 0x7f96ac100780 2023-07-14T19:23:48.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:48 smithi053 ceph-mon[96565]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:23:48.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:48 smithi053 ceph-mon[96565]: pgmap v253: 97 pgs: 97 active+clean; 615 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8.2 KiB/s wr, 19 op/s 2023-07-14T19:23:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:48 smithi070 ceph-mon[98015]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:23:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:48 smithi070 ceph-mon[98015]: pgmap v253: 97 pgs: 97 active+clean; 615 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8.2 KiB/s wr, 19 op/s 2023-07-14T19:23:48.749 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.748+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f96ac02cbd0 con 0x7f96ac100780 2023-07-14T19:23:48.753 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.752+0000 7f96a27fc700 1 -- 172.21.15.53:0/3844506567 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f96a809ad30 con 0x7f96ac100780 2023-07-14T19:23:48.753 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2023-07-14T19:23:48.754 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDEoLFkokOqLBAA8xVMLcJrv6QQgHNKWYClEw== 2023-07-14T19:23:48.755 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96980a11e0 msgr2=0x7f96980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:48.756 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96980a11e0 0x7f96980a3690 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f969c005950 tx=0x7f969c00b410 comp rx=0 tx=0).stop 2023-07-14T19:23:48.756 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 msgr2=0x7f96ac1959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:48.756 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 0x7f96ac1959f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f96a8005910 tx=0x7f96a800f710 comp rx=0 tx=0).stop 2023-07-14T19:23:48.756 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 shutdown_connections 2023-07-14T19:23:48.756 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96980a11e0 0x7f96980a3690 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.756 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96ac1059c0 0x7f96ac19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.756 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96ac100780 0x7f96ac1959f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.757 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 --2- 172.21.15.53:0/3844506567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac1030a0 0x7f96ac195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:48.757 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 >> 172.21.15.53:0/3844506567 conn(0x7f96ac0fa240 msgr2=0x7f96ac1013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:48.757 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 shutdown_connections 2023-07-14T19:23:48.757 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:48.754+0000 7f96b3279700 1 -- 172.21.15.53:0/3844506567 wait complete. 2023-07-14T19:23:48.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:48 smithi104 ceph-mon[98435]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:23:48.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:48 smithi104 ceph-mon[98435]: pgmap v253: 97 pgs: 97 active+clean; 615 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 8.2 KiB/s wr, 19 op/s 2023-07-14T19:23:49.109 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:23:49.110 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-14T19:23:49.110 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-14T19:23:49.152 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-14T19:23:49.152 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-14T19:23:49.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph mgr dump --format=json 2023-07-14T19:23:49.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:49 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3844506567' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:23:49.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:49 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3844506567' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:23:49.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:49 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3844506567' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:23:49.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:49 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3844506567' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:23:49.751 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:49.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:49 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3844506567' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:23:49.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:49 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3844506567' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:23:50.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:50 smithi053 ceph-mon[96565]: pgmap v254: 97 pgs: 97 active+clean; 616 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.6 KiB/s wr, 18 op/s 2023-07-14T19:23:50.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:50 smithi070 ceph-mon[98015]: pgmap v254: 97 pgs: 97 active+clean; 616 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.6 KiB/s wr, 18 op/s 2023-07-14T19:23:50.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:50 smithi104 ceph-mon[98435]: pgmap v254: 97 pgs: 97 active+clean; 616 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7.6 KiB/s wr, 18 op/s 2023-07-14T19:23:50.937 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 -- 172.21.15.53:0/3745514670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c1008d0 msgr2=0x7f377c1048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:50.938 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 --2- 172.21.15.53:0/3745514670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c1008d0 0x7f377c1048a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f376c0099c0 tx=0x7f376c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:23:50.938 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 -- 172.21.15.53:0/3745514670 shutdown_connections 2023-07-14T19:23:50.938 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 --2- 172.21.15.53:0/3745514670 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f377c104fd0 0x7f377c10cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:50.938 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 --2- 172.21.15.53:0/3745514670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f377c0fff30 0x7f377c100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:50.938 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 --2- 172.21.15.53:0/3745514670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c1008d0 0x7f377c1048a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:50.938 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 -- 172.21.15.53:0/3745514670 >> 172.21.15.53:0/3745514670 conn(0x7f377c0fb7d0 msgr2=0x7f377c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:50.938 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 -- 172.21.15.53:0/3745514670 shutdown_connections 2023-07-14T19:23:50.939 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 -- 172.21.15.53:0/3745514670 wait complete. 2023-07-14T19:23:50.939 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.934+0000 7f3781558700 1 Processor -- start 2023-07-14T19:23:50.939 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f3781558700 1 -- start start 2023-07-14T19:23:50.939 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f3781558700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0fff30 0x7f377c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:50.939 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f3781558700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f377c1008d0 0x7f377c19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:50.939 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f3781558700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f377c104fd0 0x7f377c195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:50.940 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f3781558700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c196160 con 0x7f377c0fff30 2023-07-14T19:23:50.940 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f3781558700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c1962a0 con 0x7f377c1008d0 2023-07-14T19:23:50.940 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f3781558700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c196460 con 0x7f377c104fd0 2023-07-14T19:23:50.940 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0fff30 0x7f377c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:50.940 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0fff30 0x7f377c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50432/0 (socket says 172.21.15.53:50432) 2023-07-14T19:23:50.940 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377affd700 1 -- 172.21.15.53:0/626171243 learned_addr learned my addr 172.21.15.53:0/626171243 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:50.941 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377b7fe700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f377c104fd0 0x7f377c195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:50.941 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377a7fc700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f377c1008d0 0x7f377c19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:50.941 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377affd700 1 -- 172.21.15.53:0/626171243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f377c104fd0 msgr2=0x7f377c195b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:50.941 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377affd700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f377c104fd0 0x7f377c195b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:50.941 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377affd700 1 -- 172.21.15.53:0/626171243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f377c1008d0 msgr2=0x7f377c19bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:50.941 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377affd700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f377c1008d0 0x7f377c19bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:50.941 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377affd700 1 -- 172.21.15.53:0/626171243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c0095f0 con 0x7f377c0fff30 2023-07-14T19:23:50.942 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.935+0000 7f377b7fe700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f377c104fd0 0x7f377c195b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:23:50.942 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.936+0000 7f377affd700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0fff30 0x7f377c19ba20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f376400d950 tx=0x7f376400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:50.942 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.936+0000 7f3773fff700 1 -- 172.21.15.53:0/626171243 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764009b40 con 0x7f377c0fff30 2023-07-14T19:23:50.942 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.936+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c196740 con 0x7f377c0fff30 2023-07-14T19:23:50.942 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.936+0000 7f3773fff700 1 -- 172.21.15.53:0/626171243 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3764009ca0 con 0x7f377c0fff30 2023-07-14T19:23:50.942 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.936+0000 7f3773fff700 1 -- 172.21.15.53:0/626171243 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376400f620 con 0x7f377c0fff30 2023-07-14T19:23:50.942 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.936+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c1a2ac0 con 0x7f377c0fff30 2023-07-14T19:23:50.943 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.937+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c04e340 con 0x7f377c0fff30 2023-07-14T19:23:50.943 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.938+0000 7f3773fff700 1 -- 172.21.15.53:0/626171243 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3764010460 con 0x7f377c0fff30 2023-07-14T19:23:50.944 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.938+0000 7f3773fff700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f37680a11e0 0x7f37680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:50.944 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.939+0000 7f377a7fc700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f37680a11e0 0x7f37680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:50.944 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.939+0000 7f3773fff700 1 -- 172.21.15.53:0/626171243 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f37640ccca0 con 0x7f377c0fff30 2023-07-14T19:23:50.944 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.939+0000 7f377a7fc700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f37680a11e0 0x7f37680a3690 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f376c0098e0 tx=0x7f376c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:50.945 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:50.943+0000 7f3773fff700 1 -- 172.21.15.53:0/626171243 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f37640d1070 con 0x7f377c0fff30 2023-07-14T19:23:51.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.196+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f377c1a2d70 con 0x7f377c0fff30 2023-07-14T19:23:51.202 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.201+0000 7f3773fff700 1 -- 172.21.15.53:0/626171243 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260125 (secure 0 0 0) 0x7f3764020090 con 0x7f377c0fff30 2023-07-14T19:23:51.205 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:51.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f37680a11e0 msgr2=0x7f37680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:51.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f37680a11e0 0x7f37680a3690 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f376c0098e0 tx=0x7f376c000bc0 comp rx=0 tx=0).stop 2023-07-14T19:23:51.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0fff30 msgr2=0x7f377c19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:51.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0fff30 0x7f377c19ba20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f376400d950 tx=0x7f376400dd10 comp rx=0 tx=0).stop 2023-07-14T19:23:51.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 shutdown_connections 2023-07-14T19:23:51.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f37680a11e0 0x7f37680a3690 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:51.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f377c104fd0 0x7f377c195b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:51.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f377c0fff30 0x7f377c19ba20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:51.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 --2- 172.21.15.53:0/626171243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f377c1008d0 0x7f377c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:51.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 >> 172.21.15.53:0/626171243 conn(0x7f377c0fb7d0 msgr2=0x7f377c0fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:51.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 shutdown_connections 2023-07-14T19:23:51.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:51.204+0000 7f3781558700 1 -- 172.21.15.53:0/626171243 wait complete. 2023-07-14T19:23:51.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:51 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/626171243' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:23:51.598 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":3014344459},{"type":"v1","addr":"172.21.15.53:6801","nonce":3014344459}]},"active_addr":"172.21.15.53:6801/3014344459","active_change":"2023-07-14T19:17:41.240221+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24103,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.53:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":2556884906}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1122088970}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1768111661}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":2283096906}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":989937245}]}]} 2023-07-14T19:23:51.606 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-14T19:23:51.607 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-14T19:23:51.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:23:51.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:51 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/626171243' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:23:51.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:51 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/626171243' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:23:52.169 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:52.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:52 smithi053 ceph-mon[96565]: pgmap v255: 97 pgs: 97 active+clean; 620 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2023-07-14T19:23:52.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:52 smithi070 ceph-mon[98015]: pgmap v255: 97 pgs: 97 active+clean; 620 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2023-07-14T19:23:52.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:52 smithi104 ceph-mon[98435]: pgmap v255: 97 pgs: 97 active+clean; 620 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2023-07-14T19:23:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 -- 172.21.15.53:0/1971665338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b810c590 msgr2=0x7f83b810e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 --2- 172.21.15.53:0/1971665338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b810c590 0x7f83b810e970 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f83b000b0a0 tx=0x7f83b000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 -- 172.21.15.53:0/1971665338 shutdown_connections 2023-07-14T19:23:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 --2- 172.21.15.53:0/1971665338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b810c590 0x7f83b810e970 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 --2- 172.21.15.53:0/1971665338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b8072960 0x7f83b8072d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 --2- 172.21.15.53:0/1971665338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83b8073270 0x7f83b810c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 -- 172.21.15.53:0/1971665338 >> 172.21.15.53:0/1971665338 conn(0x7f83b80fb790 msgr2=0x7f83b80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:53.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 -- 172.21.15.53:0/1971665338 shutdown_connections 2023-07-14T19:23:53.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 -- 172.21.15.53:0/1971665338 wait complete. 2023-07-14T19:23:53.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 Processor -- start 2023-07-14T19:23:53.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.386+0000 7f83c06ec700 1 -- start start 2023-07-14T19:23:53.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83c06ec700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83b8072960 0x7f83b8198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83c06ec700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b8073270 0x7f83b8198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83c06ec700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b810c590 0x7f83b819cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83c06ec700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b819d0f0 con 0x7f83b8073270 2023-07-14T19:23:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83c06ec700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b819d260 con 0x7f83b810c590 2023-07-14T19:23:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83c06ec700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b819d420 con 0x7f83b8072960 2023-07-14T19:23:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bdc87700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b8073270 0x7f83b8198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bdc87700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b8073270 0x7f83b8198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50446/0 (socket says 172.21.15.53:50446) 2023-07-14T19:23:53.392 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bdc87700 1 -- 172.21.15.53:0/647717124 learned_addr learned my addr 172.21.15.53:0/647717124 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:53.392 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83be488700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83b8072960 0x7f83b8198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:53.392 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bec89700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b810c590 0x7f83b819cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:53.392 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bdc87700 1 -- 172.21.15.53:0/647717124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83b8072960 msgr2=0x7f83b8198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:53.392 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bdc87700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83b8072960 0x7f83b8198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.392 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bdc87700 1 -- 172.21.15.53:0/647717124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b810c590 msgr2=0x7f83b819cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:53.392 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bdc87700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b810c590 0x7f83b819cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.393 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bdc87700 1 -- 172.21.15.53:0/647717124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b0009d00 con 0x7f83b8073270 2023-07-14T19:23:53.393 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83bdc87700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b8073270 0x7f83b8198670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f83a800d950 tx=0x7f83a800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:53.393 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83af7fe700 1 -- 172.21.15.53:0/647717124 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a800f890 con 0x7f83b8073270 2023-07-14T19:23:53.393 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b819d700 con 0x7f83b8073270 2023-07-14T19:23:53.393 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83af7fe700 1 -- 172.21.15.53:0/647717124 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83a8010460 con 0x7f83b8073270 2023-07-14T19:23:53.393 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.387+0000 7f83af7fe700 1 -- 172.21.15.53:0/647717124 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a800e610 con 0x7f83b8073270 2023-07-14T19:23:53.393 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.388+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b81a4f60 con 0x7f83b8073270 2023-07-14T19:23:53.394 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.389+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b8106eb0 con 0x7f83b8073270 2023-07-14T19:23:53.394 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.389+0000 7f83af7fe700 1 -- 172.21.15.53:0/647717124 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f83a80108d0 con 0x7f83b8073270 2023-07-14T19:23:53.394 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.390+0000 7f83af7fe700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f83a40a11e0 0x7f83a40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:53.395 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.390+0000 7f83be488700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f83a40a11e0 0x7f83a40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:53.395 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.390+0000 7f83af7fe700 1 -- 172.21.15.53:0/647717124 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f83a80cc630 con 0x7f83b8073270 2023-07-14T19:23:53.395 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.391+0000 7f83be488700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f83a40a11e0 0x7f83a40a3690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f83b4007be0 tx=0x7f83b40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:53.395 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.394+0000 7f83af7fe700 1 -- 172.21.15.53:0/647717124 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f83a8095dd0 con 0x7f83b8073270 2023-07-14T19:23:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.591+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83b802c8e0 con 0x7f83b8073270 2023-07-14T19:23:53.593 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.592+0000 7f83af7fe700 1 -- 172.21.15.53:0/647717124 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22054 (secure 0 0 0) 0x7f83a80993f0 con 0x7f83b8073270 2023-07-14T19:23:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:53.594 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":80,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:23:39.280266+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:23:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f83a40a11e0 msgr2=0x7f83a40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f83a40a11e0 0x7f83a40a3690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f83b4007be0 tx=0x7f83b40073d0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b8073270 msgr2=0x7f83b8198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b8073270 0x7f83b8198670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f83a800d950 tx=0x7f83a800dc60 comp rx=0 tx=0).stop 2023-07-14T19:23:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 shutdown_connections 2023-07-14T19:23:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f83a40a11e0 0x7f83a40a3690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83b810c590 0x7f83b819cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.597 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f83b8072960 0x7f83b8198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.597 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 --2- 172.21.15.53:0/647717124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b8073270 0x7f83b8198670 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:53.597 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 >> 172.21.15.53:0/647717124 conn(0x7f83b80fb790 msgr2=0x7f83b8103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:53.597 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.594+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 shutdown_connections 2023-07-14T19:23:53.597 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:53.595+0000 7f83c06ec700 1 -- 172.21.15.53:0/647717124 wait complete. 2023-07-14T19:23:53.717 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:53 smithi053 ceph-mon[96565]: pgmap v256: 97 pgs: 97 active+clean; 620 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.8 KiB/s wr, 14 op/s 2023-07-14T19:23:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:53 smithi070 ceph-mon[98015]: pgmap v256: 97 pgs: 97 active+clean; 620 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.8 KiB/s wr, 14 op/s 2023-07-14T19:23:53.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:53 smithi104 ceph-mon[98435]: pgmap v256: 97 pgs: 97 active+clean; 620 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.8 KiB/s wr, 14 op/s 2023-07-14T19:23:53.931 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-14T19:23:53.932 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:23:54.491 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:54.699 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:54 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/647717124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:54.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:54 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/647717124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:54.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:54 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/647717124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:55.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:55 smithi070 ceph-mon[98015]: pgmap v257: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2023-07-14T19:23:55.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 -- 172.21.15.53:0/1473070557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 msgr2=0x7fd1d410cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:55.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 --2- 172.21.15.53:0/1473070557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 0x7fd1d410cbc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00b0d0 tx=0x7fd1cc00b3e0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 -- 172.21.15.53:0/1473070557 shutdown_connections 2023-07-14T19:23:55.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 --2- 172.21.15.53:0/1473070557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 0x7fd1d410cbc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.746 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 --2- 172.21.15.53:0/1473070557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d40fe490 0x7fd1d40fe860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.746 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 --2- 172.21.15.53:0/1473070557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d40fee30 0x7fd1d4104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.746 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 -- 172.21.15.53:0/1473070557 >> 172.21.15.53:0/1473070557 conn(0x7fd1d4075280 msgr2=0x7fd1d4075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:55.746 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 -- 172.21.15.53:0/1473070557 shutdown_connections 2023-07-14T19:23:55.746 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 -- 172.21.15.53:0/1473070557 wait complete. 2023-07-14T19:23:55.746 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.742+0000 7fd1da070700 1 Processor -- start 2023-07-14T19:23:55.746 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1da070700 1 -- start start 2023-07-14T19:23:55.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1da070700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d40fe490 0x7fd1d4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:55.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1da070700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d40fee30 0x7fd1d41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:55.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1da070700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 0x7fd1d419ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:55.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1da070700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d419d020 con 0x7fd1d4105040 2023-07-14T19:23:55.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1da070700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d419d190 con 0x7fd1d40fe490 2023-07-14T19:23:55.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1da070700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d419d350 con 0x7fd1d40fee30 2023-07-14T19:23:55.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 0x7fd1d419ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:55.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 0x7fd1d419ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50458/0 (socket says 172.21.15.53:50458) 2023-07-14T19:23:55.748 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d3fff700 1 -- 172.21.15.53:0/80646955 learned_addr learned my addr 172.21.15.53:0/80646955 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:23:55.748 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d37fe700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d40fe490 0x7fd1d4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:55.748 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d2ffd700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d40fee30 0x7fd1d41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:55.748 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d3fff700 1 -- 172.21.15.53:0/80646955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d40fee30 msgr2=0x7fd1d41985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:55.748 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d3fff700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d40fee30 0x7fd1d41985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.748 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d3fff700 1 -- 172.21.15.53:0/80646955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d40fe490 msgr2=0x7fd1d4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:55.748 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d3fff700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d40fe490 0x7fd1d4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.749 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d3fff700 1 -- 172.21.15.53:0/80646955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc009d00 con 0x7fd1d4105040 2023-07-14T19:23:55.749 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d2ffd700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d40fee30 0x7fd1d41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:23:55.749 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.743+0000 7fd1d3fff700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 0x7fd1d419ca10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc007870 tx=0x7fd1cc0078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:55.749 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.744+0000 7fd1d0ff9700 1 -- 172.21.15.53:0/80646955 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc00e040 con 0x7fd1d4105040 2023-07-14T19:23:55.749 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.744+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d419d5d0 con 0x7fd1d4105040 2023-07-14T19:23:55.749 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.744+0000 7fd1d0ff9700 1 -- 172.21.15.53:0/80646955 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1cc003e10 con 0x7fd1d4105040 2023-07-14T19:23:55.749 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.744+0000 7fd1d0ff9700 1 -- 172.21.15.53:0/80646955 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc01b980 con 0x7fd1d4105040 2023-07-14T19:23:55.749 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.744+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d41a4e90 con 0x7fd1d4105040 2023-07-14T19:23:55.750 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.745+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d404e340 con 0x7fd1d4105040 2023-07-14T19:23:55.750 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.746+0000 7fd1d0ff9700 1 -- 172.21.15.53:0/80646955 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd1cc024410 con 0x7fd1d4105040 2023-07-14T19:23:55.750 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.746+0000 7fd1d0ff9700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd1bc0a11e0 0x7fd1bc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:55.751 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.747+0000 7fd1d0ff9700 1 -- 172.21.15.53:0/80646955 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd1cc0cdf00 con 0x7fd1d4105040 2023-07-14T19:23:55.751 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.747+0000 7fd1d37fe700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd1bc0a11e0 0x7fd1bc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:55.751 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.747+0000 7fd1d37fe700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd1bc0a11e0 0x7fd1bc0a3690 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400ac50 tx=0x7fd1c400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:55.752 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.751+0000 7fd1d0ff9700 1 -- 172.21.15.53:0/80646955 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd1cc097730 con 0x7fd1d4105040 2023-07-14T19:23:55.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:55 smithi104 ceph-mon[98435]: pgmap v257: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2023-07-14T19:23:55.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:55 smithi053 ceph-mon[96565]: pgmap v257: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2023-07-14T19:23:55.950 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.949+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1d41a5020 con 0x7fd1d4105040 2023-07-14T19:23:55.951 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.950+0000 7fd1d0ff9700 1 -- 172.21.15.53:0/80646955 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22054 (secure 0 0 0) 0x7fd1cc09ad50 con 0x7fd1d4105040 2023-07-14T19:23:55.952 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:23:55.952 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":80,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:23:39.280266+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:23:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd1bc0a11e0 msgr2=0x7fd1bc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd1bc0a11e0 0x7fd1bc0a3690 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400ac50 tx=0x7fd1c400a380 comp rx=0 tx=0).stop 2023-07-14T19:23:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 msgr2=0x7fd1d419ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 0x7fd1d419ca10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc007870 tx=0x7fd1cc0078a0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 shutdown_connections 2023-07-14T19:23:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd1bc0a11e0 0x7fd1bc0a3690 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d4105040 0x7fd1d419ca10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1d40fe490 0x7fd1d4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 --2- 172.21.15.53:0/80646955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd1d40fee30 0x7fd1d41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 >> 172.21.15.53:0/80646955 conn(0x7fd1d4075280 msgr2=0x7fd1d40fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 shutdown_connections 2023-07-14T19:23:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:23:55.952+0000 7fd1da070700 1 -- 172.21.15.53:0/80646955 wait complete. 2023-07-14T19:23:56.335 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-14T19:23:56.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-14T19:23:56.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-14T19:23:56.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-14T19:23:56.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-14T19:23:56.338 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-14T19:23:56.338 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-14T19:23:56.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-14T19:23:56.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-07-14T19:23:56.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-07-14T19:23:56.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-07-14T19:23:56.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-07-14T19:23:56.574 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:56 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/80646955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:56.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:56 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/80646955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:56.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:56 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/80646955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:23:57.711 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:57.726 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:57.862 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:57.867 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:58.061 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:58.271 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:58.559 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:58 smithi053 ceph-mon[96565]: pgmap v258: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2023-07-14T19:23:58.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:58 smithi070 ceph-mon[98015]: pgmap v258: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2023-07-14T19:23:58.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:58 smithi104 ceph-mon[98435]: pgmap v258: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2023-07-14T19:23:58.957 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:58.965 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:58.969 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:59.453 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:59.465 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:59.540 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:23:59 smithi053 ceph-mon[96565]: pgmap v259: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2023-07-14T19:23:59.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:23:59 smithi070 ceph-mon[98015]: pgmap v259: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2023-07-14T19:23:59.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:23:59 smithi104 ceph-mon[98435]: pgmap v259: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2023-07-14T19:24:01.472 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:01 smithi053 ceph-mon[96565]: pgmap v260: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:24:01.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:01 smithi070 ceph-mon[98015]: pgmap v260: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:24:01.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:01 smithi104 ceph-mon[98435]: pgmap v260: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:24:02.908 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.906+0000 7ff3e25b9700 1 -- 172.21.15.53:0/49965789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3dc0feef0 msgr2=0x7ff3dc104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:02.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.906+0000 7ff3e25b9700 1 --2- 172.21.15.53:0/49965789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3dc0feef0 0x7ff3dc104960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00099c0 tx=0x7ff3d0009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:02.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.906+0000 7ff3e25b9700 1 -- 172.21.15.53:0/49965789 shutdown_connections 2023-07-14T19:24:02.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.906+0000 7ff3e25b9700 1 --2- 172.21.15.53:0/49965789 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3dc105090 0x7ff3dc10cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:02.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.906+0000 7ff3e25b9700 1 --2- 172.21.15.53:0/49965789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc0fe550 0x7ff3dc0fe920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:02.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.906+0000 7ff3e25b9700 1 --2- 172.21.15.53:0/49965789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3dc0feef0 0x7ff3dc104960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:02.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.906+0000 7ff3e25b9700 1 -- 172.21.15.53:0/49965789 >> 172.21.15.53:0/49965789 conn(0x7ff3dc0751e0 msgr2=0x7ff3dc0755e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:02.909 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.906+0000 7ff3e25b9700 1 -- 172.21.15.53:0/49965789 shutdown_connections 2023-07-14T19:24:02.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.906+0000 7ff3e25b9700 1 -- 172.21.15.53:0/49965789 wait complete. 2023-07-14T19:24:02.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e25b9700 1 Processor -- start 2023-07-14T19:24:02.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e25b9700 1 -- start start 2023-07-14T19:24:02.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e25b9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc0fe550 0x7ff3dc0ff4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:02.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e25b9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3dc0feef0 0x7ff3dc0ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:02.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e25b9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3dc105090 0x7ff3dc0fff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:02.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e25b9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc1a4160 con 0x7ff3dc105090 2023-07-14T19:24:02.910 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e25b9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc1a42d0 con 0x7ff3dc0fe550 2023-07-14T19:24:02.911 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e25b9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc1a4490 con 0x7ff3dc0feef0 2023-07-14T19:24:02.912 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e0b56700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3dc105090 0x7ff3dc0fff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:02.912 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e0b56700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3dc105090 0x7ff3dc0fff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50470/0 (socket says 172.21.15.53:50470) 2023-07-14T19:24:02.912 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.907+0000 7ff3e0b56700 1 -- 172.21.15.53:0/4105791231 learned_addr learned my addr 172.21.15.53:0/4105791231 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:02.912 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3db7fe700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3dc0feef0 0x7ff3dc0ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:02.912 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3dbfff700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc0fe550 0x7ff3dc0ff4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:02.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3e0b56700 1 -- 172.21.15.53:0/4105791231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3dc0feef0 msgr2=0x7ff3dc0ffa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:02.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3e0b56700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3dc0feef0 0x7ff3dc0ffa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:02.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3e0b56700 1 -- 172.21.15.53:0/4105791231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc0fe550 msgr2=0x7ff3dc0ff4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:02.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3e0b56700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc0fe550 0x7ff3dc0ff4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:02.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3e0b56700 1 -- 172.21.15.53:0/4105791231 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d00095f0 con 0x7ff3dc105090 2023-07-14T19:24:02.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3db7fe700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3dc0feef0 0x7ff3dc0ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:02.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3e0b56700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3dc105090 0x7ff3dc0fff90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400ef30 tx=0x7ff3d400c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:02.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3d97fa700 1 -- 172.21.15.53:0/4105791231 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4018070 con 0x7ff3dc105090 2023-07-14T19:24:02.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3d97fa700 1 -- 172.21.15.53:0/4105791231 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3d400f040 con 0x7ff3dc105090 2023-07-14T19:24:02.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc1a4710 con 0x7ff3dc105090 2023-07-14T19:24:02.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3d97fa700 1 -- 172.21.15.53:0/4105791231 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4003dc0 con 0x7ff3dc105090 2023-07-14T19:24:02.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.908+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc1a4b90 con 0x7ff3dc105090 2023-07-14T19:24:02.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.909+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff3dc04e340 con 0x7ff3dc105090 2023-07-14T19:24:02.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.911+0000 7ff3d97fa700 1 -- 172.21.15.53:0/4105791231 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff3d400caa0 con 0x7ff3dc105090 2023-07-14T19:24:02.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.911+0000 7ff3d97fa700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff3c40a11e0 0x7ff3c40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:02.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.911+0000 7ff3dbfff700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff3c40a11e0 0x7ff3c40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:02.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.911+0000 7ff3d97fa700 1 -- 172.21.15.53:0/4105791231 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff3d40ccc20 con 0x7ff3dc105090 2023-07-14T19:24:02.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.911+0000 7ff3d97fa700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] conn(0x7ff3c40aa9b0 0x7ff3c40acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:02.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.911+0000 7ff3d97fa700 1 -- 172.21.15.53:0/4105791231 --> [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3c40ad470 con 0x7ff3c40aa9b0 2023-07-14T19:24:02.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.911+0000 7ff3d97fa700 1 -- 172.21.15.53:0/4105791231 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3d40cd0c0 con 0x7ff3dc105090 2023-07-14T19:24:02.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.912+0000 7ff3dbfff700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff3c40a11e0 0x7ff3c40a3690 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc005ea0 tx=0x7ff3cc005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:02.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.912+0000 7ff3db7fe700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] conn(0x7ff3c40aa9b0 0x7ff3c40acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:02.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.912+0000 7ff3db7fe700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] conn(0x7ff3c40aa9b0 0x7ff3c40acdc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:02.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.913+0000 7ff3d97fa700 1 -- 172.21.15.53:0/4105791231 <== osd.10 v2:172.21.15.70:6824/1946322503 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff3c40ad470 con 0x7ff3c40aa9b0 2023-07-14T19:24:02.930 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.929+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 --> [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff3dc10ff70 con 0x7ff3c40aa9b0 2023-07-14T19:24:02.931 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3d97fa700 1 -- 172.21.15.53:0/4105791231 <== osd.10 v2:172.21.15.70:6824/1946322503 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff3d000bb30 con 0x7ff3c40aa9b0 2023-07-14T19:24:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 >> [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] conn(0x7ff3c40aa9b0 msgr2=0x7ff3c40acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] conn(0x7ff3c40aa9b0 0x7ff3c40acdc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff3c40a11e0 msgr2=0x7ff3c40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff3c40a11e0 0x7ff3c40a3690 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc005ea0 tx=0x7ff3cc005e30 comp rx=0 tx=0).stop 2023-07-14T19:24:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3dc105090 msgr2=0x7ff3dc0fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 --2- 172.21.15.53:0/4105791231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3dc105090 0x7ff3dc0fff90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400ef30 tx=0x7ff3d400c450 comp rx=0 tx=0).stop 2023-07-14T19:24:02.933 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e0b56700 1 -- 172.21.15.53:0/4105791231 reap_dead start 2023-07-14T19:24:02.933 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 shutdown_connections 2023-07-14T19:24:02.933 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 >> 172.21.15.53:0/4105791231 conn(0x7ff3dc0751e0 msgr2=0x7ff3dc103940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:02.933 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 shutdown_connections 2023-07-14T19:24:02.933 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:02.930+0000 7ff3e25b9700 1 -- 172.21.15.53:0/4105791231 wait complete. 2023-07-14T19:24:03.509 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.465+0000 7fb7958ed700 1 -- 172.21.15.53:0/488092677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb790100f20 msgr2=0x7fb7901012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:03.509 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.465+0000 7fb7958ed700 1 --2- 172.21.15.53:0/488092677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb790100f20 0x7fb7901012f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb784009990 tx=0x7fb784009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.510 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.465+0000 7fb7958ed700 1 -- 172.21.15.53:0/488092677 shutdown_connections 2023-07-14T19:24:03.510 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.466+0000 7fb7958ed700 1 --2- 172.21.15.53:0/488092677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb79010a270 0x7fb79010c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.510 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.466+0000 7fb7958ed700 1 --2- 172.21.15.53:0/488092677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb790100f20 0x7fb7901012f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.511 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.466+0000 7fb7958ed700 1 --2- 172.21.15.53:0/488092677 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb790101830 0x7fb790109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.511 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.466+0000 7fb7958ed700 1 -- 172.21.15.53:0/488092677 >> 172.21.15.53:0/488092677 conn(0x7fb790075ca0 msgr2=0x7fb7900760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:03.511 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.466+0000 7fb7958ed700 1 -- 172.21.15.53:0/488092677 shutdown_connections 2023-07-14T19:24:03.511 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.466+0000 7fb7958ed700 1 -- 172.21.15.53:0/488092677 wait complete. 2023-07-14T19:24:03.511 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.466+0000 7fb7958ed700 1 Processor -- start 2023-07-14T19:24:03.512 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.466+0000 7fb7958ed700 1 -- start start 2023-07-14T19:24:03.512 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.466+0000 7fb7958ed700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb790100f20 0x7fb790195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.512 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7958ed700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb790101830 0x7fb7901962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.512 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7958ed700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb79010a270 0x7fb79019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.512 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7958ed700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79019ad70 con 0x7fb790100f20 2023-07-14T19:24:03.513 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7958ed700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79019aee0 con 0x7fb79010a270 2023-07-14T19:24:03.513 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7958ed700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79019b0a0 con 0x7fb790101830 2023-07-14T19:24:03.513 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb790100f20 0x7fb790195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.513 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb790100f20 0x7fb790195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50482/0 (socket says 172.21.15.53:50482) 2023-07-14T19:24:03.514 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78effd700 1 -- 172.21.15.53:0/1476598331 learned_addr learned my addr 172.21.15.53:0/1476598331 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:03.514 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78e7fc700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb790101830 0x7fb7901962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.514 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78f7fe700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb79010a270 0x7fb79019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.514 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78effd700 1 -- 172.21.15.53:0/1476598331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb790101830 msgr2=0x7fb7901962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:03.514 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78effd700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb790101830 0x7fb7901962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.514 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78effd700 1 -- 172.21.15.53:0/1476598331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb79010a270 msgr2=0x7fb79019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78effd700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb79010a270 0x7fb79019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78effd700 1 -- 172.21.15.53:0/1476598331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7840095f0 con 0x7fb790100f20 2023-07-14T19:24:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78e7fc700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb790101830 0x7fb7901962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb78effd700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb790100f20 0x7fb790195db0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb7840095c0 tx=0x7fb78400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7948eb700 1 -- 172.21.15.53:0/1476598331 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78401d070 con 0x7fb790100f20 2023-07-14T19:24:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb79019b320 con 0x7fb790100f20 2023-07-14T19:24:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7948eb700 1 -- 172.21.15.53:0/1476598331 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb78400fc00 con 0x7fb790100f20 2023-07-14T19:24:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7948eb700 1 -- 172.21.15.53:0/1476598331 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784017650 con 0x7fb790100f20 2023-07-14T19:24:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.467+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7901a2be0 con 0x7fb790100f20 2023-07-14T19:24:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.468+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb79004e340 con 0x7fb790100f20 2023-07-14T19:24:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.469+0000 7fb7948eb700 1 -- 172.21.15.53:0/1476598331 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb784022470 con 0x7fb790100f20 2023-07-14T19:24:03.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.470+0000 7fb7948eb700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb7780a11e0 0x7fb7780a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.470+0000 7fb7948eb700 1 -- 172.21.15.53:0/1476598331 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb7840ce490 con 0x7fb790100f20 2023-07-14T19:24:03.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.470+0000 7fb78e7fc700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb7780a11e0 0x7fb7780a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.470+0000 7fb7948eb700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] conn(0x7fb7780aa9b0 0x7fb7780acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.470+0000 7fb7948eb700 1 -- 172.21.15.53:0/1476598331 --> [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7780ad470 con 0x7fb7780aa9b0 2023-07-14T19:24:03.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.470+0000 7fb7948eb700 1 -- 172.21.15.53:0/1476598331 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7840ce930 con 0x7fb790100f20 2023-07-14T19:24:03.517 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.471+0000 7fb78e7fc700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb7780a11e0 0x7fb7780a3690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb780005950 tx=0x7fb7800058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:03.518 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.471+0000 7fb78f7fe700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] conn(0x7fb7780aa9b0 0x7fb7780acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.518 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.471+0000 7fb78f7fe700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] conn(0x7fb7780aa9b0 0x7fb7780acdc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:03.518 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.472+0000 7fb7948eb700 1 -- 172.21.15.53:0/1476598331 <== osd.8 v2:172.21.15.104:6816/48256695 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb7780ad470 con 0x7fb7780aa9b0 2023-07-14T19:24:03.519 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.488+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 --> [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb79010fed0 con 0x7fb7780aa9b0 2023-07-14T19:24:03.519 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7948eb700 1 -- 172.21.15.53:0/1476598331 <== osd.8 v2:172.21.15.104:6816/48256695 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb788009b10 con 0x7fb7780aa9b0 2023-07-14T19:24:03.519 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 >> [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] conn(0x7fb7780aa9b0 msgr2=0x7fb7780acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:03.519 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] conn(0x7fb7780aa9b0 0x7fb7780acdc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.519 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb7780a11e0 msgr2=0x7fb7780a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:03.520 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb7780a11e0 0x7fb7780a3690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb780005950 tx=0x7fb7800058e0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.520 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb790100f20 msgr2=0x7fb790195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:03.520 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 --2- 172.21.15.53:0/1476598331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb790100f20 0x7fb790195db0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb7840095c0 tx=0x7fb78400f710 comp rx=0 tx=0).stop 2023-07-14T19:24:03.521 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb78f7fe700 1 -- 172.21.15.53:0/1476598331 reap_dead start 2023-07-14T19:24:03.521 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 shutdown_connections 2023-07-14T19:24:03.521 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 >> 172.21.15.53:0/1476598331 conn(0x7fb790075ca0 msgr2=0x7fb7900fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:03.521 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 shutdown_connections 2023-07-14T19:24:03.521 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.489+0000 7fb7958ed700 1 -- 172.21.15.53:0/1476598331 wait complete. 2023-07-14T19:24:03.522 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:03 smithi053 ceph-mon[96565]: pgmap v261: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:24:03.605 INFO:teuthology.orchestra.run.smithi053.stdout:292057776147 2023-07-14T19:24:03.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-07-14T19:24:03.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:03 smithi070 ceph-mon[98015]: pgmap v261: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:24:03.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:03 smithi104 ceph-mon[98435]: pgmap v261: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:24:03.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.983+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1632405613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff54105020 msgr2=0x7eff5410cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:03.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.983+0000 7eff5a0e6700 1 --2- 172.21.15.53:0/1632405613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff54105020 0x7eff5410cba0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0099c0 tx=0x7eff3c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.983+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1632405613 shutdown_connections 2023-07-14T19:24:03.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.983+0000 7eff5a0e6700 1 --2- 172.21.15.53:0/1632405613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff54105020 0x7eff5410cba0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.983+0000 7eff5a0e6700 1 --2- 172.21.15.53:0/1632405613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff540fff80 0x7eff54100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.983+0000 7eff5a0e6700 1 --2- 172.21.15.53:0/1632405613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff54100920 0x7eff541048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.983+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1632405613 >> 172.21.15.53:0/1632405613 conn(0x7eff540fb840 msgr2=0x7eff540fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:03.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.983+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1632405613 shutdown_connections 2023-07-14T19:24:03.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.983+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1632405613 wait complete. 2023-07-14T19:24:03.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff5a0e6700 1 Processor -- start 2023-07-14T19:24:03.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff5a0e6700 1 -- start start 2023-07-14T19:24:03.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff5a0e6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff540fff80 0x7eff54195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff5a0e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff54100920 0x7eff54196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff5a0e6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff54105020 0x7eff5419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff5a0e6700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5419ad00 con 0x7eff540fff80 2023-07-14T19:24:03.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff5a0e6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5419ae70 con 0x7eff54100920 2023-07-14T19:24:03.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff5a0e6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5419b030 con 0x7eff54105020 2023-07-14T19:24:03.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff537fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff540fff80 0x7eff54195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff537fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff540fff80 0x7eff54195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50496/0 (socket says 172.21.15.53:50496) 2023-07-14T19:24:03.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff537fe700 1 -- 172.21.15.53:0/1781908678 learned_addr learned my addr 172.21.15.53:0/1781908678 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:03.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff52ffd700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff54100920 0x7eff54196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff53fff700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff54105020 0x7eff5419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff537fe700 1 -- 172.21.15.53:0/1781908678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff54105020 msgr2=0x7eff5419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:03.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff537fe700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff54105020 0x7eff5419a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff537fe700 1 -- 172.21.15.53:0/1781908678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff54100920 msgr2=0x7eff54196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:03.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff537fe700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff54100920 0x7eff54196280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:03.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.984+0000 7eff537fe700 1 -- 172.21.15.53:0/1781908678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c0095f0 con 0x7eff540fff80 2023-07-14T19:24:03.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.985+0000 7eff537fe700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff540fff80 0x7eff54195d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7eff4400ef10 tx=0x7eff4400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:03.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.985+0000 7eff50ff9700 1 -- 172.21.15.53:0/1781908678 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4400ca70 con 0x7eff540fff80 2023-07-14T19:24:03.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.985+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5419b2b0 con 0x7eff540fff80 2023-07-14T19:24:03.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.985+0000 7eff50ff9700 1 -- 172.21.15.53:0/1781908678 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff4400f040 con 0x7eff540fff80 2023-07-14T19:24:03.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.985+0000 7eff50ff9700 1 -- 172.21.15.53:0/1781908678 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff44014480 con 0x7eff540fff80 2023-07-14T19:24:03.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.985+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff541a2b70 con 0x7eff540fff80 2023-07-14T19:24:03.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.986+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7eff5410fec0 con 0x7eff540fff80 2023-07-14T19:24:03.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.987+0000 7eff50ff9700 1 -- 172.21.15.53:0/1781908678 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7eff44007500 con 0x7eff540fff80 2023-07-14T19:24:03.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.988+0000 7eff50ff9700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7eff400a11e0 0x7eff400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.988+0000 7eff52ffd700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7eff400a11e0 0x7eff400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.988+0000 7eff50ff9700 1 -- 172.21.15.53:0/1781908678 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7eff440cd710 con 0x7eff540fff80 2023-07-14T19:24:03.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.988+0000 7eff50ff9700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] conn(0x7eff400aa9b0 0x7eff400acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:03.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.988+0000 7eff50ff9700 1 -- 172.21.15.53:0/1781908678 --> [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff400ad470 con 0x7eff400aa9b0 2023-07-14T19:24:03.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.988+0000 7eff50ff9700 1 -- 172.21.15.53:0/1781908678 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff440cdbb0 con 0x7eff540fff80 2023-07-14T19:24:03.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.988+0000 7eff53fff700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] conn(0x7eff400aa9b0 0x7eff400acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:03.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.988+0000 7eff52ffd700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7eff400a11e0 0x7eff400a3690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7eff54197360 tx=0x7eff4c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:03.993 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.988+0000 7eff53fff700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] conn(0x7eff400aa9b0 0x7eff400acdc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:03.993 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:03.989+0000 7eff50ff9700 1 -- 172.21.15.53:0/1781908678 <== osd.3 v2:172.21.15.53:6810/1549618174 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7eff400ad470 con 0x7eff400aa9b0 2023-07-14T19:24:04.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.005+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 --> [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7eff5404e340 con 0x7eff400aa9b0 2023-07-14T19:24:04.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.005+0000 7eff50ff9700 1 -- 172.21.15.53:0/1781908678 <== osd.3 v2:172.21.15.53:6810/1549618174 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7eff5404e340 con 0x7eff400aa9b0 2023-07-14T19:24:04.007 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.005+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] conn(0x7eff400aa9b0 msgr2=0x7eff400acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:04.007 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.005+0000 7eff5a0e6700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] conn(0x7eff400aa9b0 0x7eff400acdc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:04.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.005+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7eff400a11e0 msgr2=0x7eff400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:04.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.005+0000 7eff5a0e6700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7eff400a11e0 0x7eff400a3690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7eff54197360 tx=0x7eff4c00b410 comp rx=0 tx=0).stop 2023-07-14T19:24:04.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.005+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff540fff80 msgr2=0x7eff54195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:04.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.006+0000 7eff5a0e6700 1 --2- 172.21.15.53:0/1781908678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff540fff80 0x7eff54195d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7eff4400ef10 tx=0x7eff4400c3a0 comp rx=0 tx=0).stop 2023-07-14T19:24:04.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.006+0000 7eff53fff700 1 -- 172.21.15.53:0/1781908678 reap_dead start 2023-07-14T19:24:04.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.006+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 shutdown_connections 2023-07-14T19:24:04.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.006+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 >> 172.21.15.53:0/1781908678 conn(0x7eff540fb840 msgr2=0x7eff540fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:04.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.006+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 shutdown_connections 2023-07-14T19:24:04.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.006+0000 7eff5a0e6700 1 -- 172.21.15.53:0/1781908678 wait complete. 2023-07-14T19:24:05.102 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.902+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/1703770373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e041008d0 msgr2=0x7f7e041048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.103 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.902+0000 7f7e0b7e8700 1 --2- 172.21.15.53:0/1703770373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e041008d0 0x7f7e041048a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7df40099c0 tx=0x7f7df4009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.103 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/1703770373 shutdown_connections 2023-07-14T19:24:05.103 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 --2- 172.21.15.53:0/1703770373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e04104fd0 0x7f7e0410cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.103 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 --2- 172.21.15.53:0/1703770373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e040fff30 0x7f7e04100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.103 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 --2- 172.21.15.53:0/1703770373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e041008d0 0x7f7e041048a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.103 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/1703770373 >> 172.21.15.53:0/1703770373 conn(0x7f7e040fb790 msgr2=0x7f7e040fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:05.104 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/1703770373 shutdown_connections 2023-07-14T19:24:05.104 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/1703770373 wait complete. 2023-07-14T19:24:05.104 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 Processor -- start 2023-07-14T19:24:05.104 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 -- start start 2023-07-14T19:24:05.104 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e040fff30 0x7f7e04195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.104 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e041008d0 0x7f7e04196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.104 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.903+0000 7f7e0b7e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e04104fd0 0x7f7e0419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.105 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e0b7e8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0419ad40 con 0x7f7e040fff30 2023-07-14T19:24:05.105 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e0b7e8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0419aeb0 con 0x7f7e041008d0 2023-07-14T19:24:05.105 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e0b7e8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0419b070 con 0x7f7e04104fd0 2023-07-14T19:24:05.105 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09584700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e040fff30 0x7f7e04195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.105 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09584700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e040fff30 0x7f7e04195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50510/0 (socket says 172.21.15.53:50510) 2023-07-14T19:24:05.105 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09584700 1 -- 172.21.15.53:0/2417650621 learned_addr learned my addr 172.21.15.53:0/2417650621 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:05.105 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e08d83700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e041008d0 0x7f7e04196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.105 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09d85700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e04104fd0 0x7f7e0419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.106 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09584700 1 -- 172.21.15.53:0/2417650621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e04104fd0 msgr2=0x7f7e0419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.106 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09584700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e04104fd0 0x7f7e0419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.106 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09584700 1 -- 172.21.15.53:0/2417650621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e041008d0 msgr2=0x7f7e04196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.106 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09584700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e041008d0 0x7f7e04196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.106 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09584700 1 -- 172.21.15.53:0/2417650621 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df40095f0 con 0x7f7e040fff30 2023-07-14T19:24:05.106 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09d85700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e04104fd0 0x7f7e0419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:05.106 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e09584700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e040fff30 0x7f7e04195dd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000c920 tx=0x7f7e0000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:05.107 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7dfa7fc700 1 -- 172.21.15.53:0/2417650621 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00007b00 con 0x7f7e040fff30 2023-07-14T19:24:05.107 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7dfa7fc700 1 -- 172.21.15.53:0/2417650621 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e0000ce70 con 0x7f7e040fff30 2023-07-14T19:24:05.107 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7dfa7fc700 1 -- 172.21.15.53:0/2417650621 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e000054e0 con 0x7f7e040fff30 2023-07-14T19:24:05.107 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0419b350 con 0x7f7e040fff30 2023-07-14T19:24:05.107 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.904+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e041a2bb0 con 0x7f7e040fff30 2023-07-14T19:24:05.107 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.905+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e0404e340 con 0x7f7e040fff30 2023-07-14T19:24:05.107 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.907+0000 7f7dfa7fc700 1 -- 172.21.15.53:0/2417650621 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7e00004750 con 0x7f7e040fff30 2023-07-14T19:24:05.108 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.907+0000 7f7dfa7fc700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7df00a1260 0x7f7df00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.108 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.907+0000 7f7e08d83700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7df00a1260 0x7f7df00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.108 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.908+0000 7f7dfa7fc700 1 -- 172.21.15.53:0/2417650621 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7e000cc850 con 0x7f7e040fff30 2023-07-14T19:24:05.109 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.908+0000 7f7dfa7fc700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] conn(0x7f7df00aaa30 0x7f7df00ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.109 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.908+0000 7f7dfa7fc700 1 -- 172.21.15.53:0/2417650621 --> [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7df00ad4f0 con 0x7f7df00aaa30 2023-07-14T19:24:05.109 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.908+0000 7f7dfa7fc700 1 -- 172.21.15.53:0/2417650621 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e000cccf0 con 0x7f7e040fff30 2023-07-14T19:24:05.109 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.908+0000 7f7e09d85700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] conn(0x7f7df00aaa30 0x7f7df00ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.109 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.908+0000 7f7e08d83700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7df00a1260 0x7f7df00a3710 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7df400b580 tx=0x7f7df4003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:05.109 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.908+0000 7f7e09d85700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] conn(0x7f7df00aaa30 0x7f7df00ace40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:05.109 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.909+0000 7f7dfa7fc700 1 -- 172.21.15.53:0/2417650621 <== osd.6 v2:172.21.15.53:6818/4148952318 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f7df00ad4f0 con 0x7f7df00aaa30 2023-07-14T19:24:05.110 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.926+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 --> [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7e0410fdf0 con 0x7f7df00aaa30 2023-07-14T19:24:05.110 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.926+0000 7f7dfa7fc700 1 -- 172.21.15.53:0/2417650621 <== osd.6 v2:172.21.15.53:6818/4148952318 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7e0410fdf0 con 0x7f7df00aaa30 2023-07-14T19:24:05.110 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.926+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] conn(0x7f7df00aaa30 msgr2=0x7f7df00ace40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.110 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.926+0000 7f7e0b7e8700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] conn(0x7f7df00aaa30 0x7f7df00ace40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.110 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.927+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7df00a1260 msgr2=0x7f7df00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.111 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.927+0000 7f7e0b7e8700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7df00a1260 0x7f7df00a3710 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7df400b580 tx=0x7f7df4003680 comp rx=0 tx=0).stop 2023-07-14T19:24:05.111 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.927+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e040fff30 msgr2=0x7f7e04195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.111 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.927+0000 7f7e0b7e8700 1 --2- 172.21.15.53:0/2417650621 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e040fff30 0x7f7e04195dd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000c920 tx=0x7f7e0000cc30 comp rx=0 tx=0).stop 2023-07-14T19:24:05.111 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.927+0000 7f7e09d85700 1 -- 172.21.15.53:0/2417650621 reap_dead start 2023-07-14T19:24:05.111 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.927+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 shutdown_connections 2023-07-14T19:24:05.111 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.927+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 >> 172.21.15.53:0/2417650621 conn(0x7f7e040fb790 msgr2=0x7f7e040fc230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:05.111 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.927+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 shutdown_connections 2023-07-14T19:24:05.112 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:04.927+0000 7f7e0b7e8700 1 -- 172.21.15.53:0/2417650621 wait complete. 2023-07-14T19:24:05.473 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:05 smithi053 ceph-mon[96565]: pgmap v262: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:24:05.664 INFO:teuthology.orchestra.run.smithi053.stdout:103079215153 2023-07-14T19:24:05.665 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-14T19:24:05.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:05 smithi070 ceph-mon[98015]: pgmap v262: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:24:05.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:05 smithi104 ceph-mon[98435]: pgmap v262: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:24:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.847+0000 7fc13a5a9700 1 -- 172.21.15.53:0/2816995762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340febc0 msgr2=0x7fc13410cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.847+0000 7fc13a5a9700 1 --2- 172.21.15.53:0/2816995762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340febc0 0x7fc13410cb30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00b0a0 tx=0x7fc12c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.847+0000 7fc13a5a9700 1 -- 172.21.15.53:0/2816995762 shutdown_connections 2023-07-14T19:24:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.847+0000 7fc13a5a9700 1 --2- 172.21.15.53:0/2816995762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340febc0 0x7fc13410cb30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.847+0000 7fc13a5a9700 1 --2- 172.21.15.53:0/2816995762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc134104c70 0x7fc134105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.847+0000 7fc13a5a9700 1 --2- 172.21.15.53:0/2816995762 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1340fe210 0x7fc1340fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.847+0000 7fc13a5a9700 1 -- 172.21.15.53:0/2816995762 >> 172.21.15.53:0/2816995762 conn(0x7fc134075280 msgr2=0x7fc134075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.847+0000 7fc13a5a9700 1 -- 172.21.15.53:0/2816995762 shutdown_connections 2023-07-14T19:24:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.847+0000 7fc13a5a9700 1 -- 172.21.15.53:0/2816995762 wait complete. 2023-07-14T19:24:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.848+0000 7fc13a5a9700 1 Processor -- start 2023-07-14T19:24:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.848+0000 7fc13a5a9700 1 -- start start 2023-07-14T19:24:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc13a5a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340fe210 0x7fc1341a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc13a5a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1340febc0 0x7fc1341a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc13a5a9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc134104c70 0x7fc13410b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc13a5a9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13410bec0 con 0x7fc134104c70 2023-07-14T19:24:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc13a5a9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13410c030 con 0x7fc1340fe210 2023-07-14T19:24:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc13a5a9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13410c1f0 con 0x7fc1340febc0 2023-07-14T19:24:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc138b46700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc134104c70 0x7fc13410b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc138b46700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc134104c70 0x7fc13410b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50544/0 (socket says 172.21.15.53:50544) 2023-07-14T19:24:05.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc138b46700 1 -- 172.21.15.53:0/696271819 learned_addr learned my addr 172.21.15.53:0/696271819 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:05.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc133fff700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340fe210 0x7fc1341a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc1337fe700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1340febc0 0x7fc1341a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.854 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc138b46700 1 -- 172.21.15.53:0/696271819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1340febc0 msgr2=0x7fc1341a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.854 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc138b46700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1340febc0 0x7fc1341a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.854 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc138b46700 1 -- 172.21.15.53:0/696271819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340fe210 msgr2=0x7fc1341a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.854 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc138b46700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1340fe210 0x7fc1341a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc138b46700 1 -- 172.21.15.53:0/696271819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c009d00 con 0x7fc134104c70 2023-07-14T19:24:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.849+0000 7fc138b46700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc134104c70 0x7fc13410b810 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009fd0 tx=0x7fc12c00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.850+0000 7fc1317fa700 1 -- 172.21.15.53:0/696271819 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c00e040 con 0x7fc134104c70 2023-07-14T19:24:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.850+0000 7fc1317fa700 1 -- 172.21.15.53:0/696271819 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc12c004e60 con 0x7fc134104c70 2023-07-14T19:24:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.850+0000 7fc1317fa700 1 -- 172.21.15.53:0/696271819 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c005280 con 0x7fc134104c70 2023-07-14T19:24:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.850+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13410c470 con 0x7fc134104c70 2023-07-14T19:24:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.850+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1341a9300 con 0x7fc134104c70 2023-07-14T19:24:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.851+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc134110010 con 0x7fc134104c70 2023-07-14T19:24:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.853+0000 7fc1317fa700 1 -- 172.21.15.53:0/696271819 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc12c01d470 con 0x7fc134104c70 2023-07-14T19:24:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.854+0000 7fc1317fa700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc11c0a11e0 0x7fc11c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.854+0000 7fc1317fa700 1 -- 172.21.15.53:0/696271819 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc12c0ce2c0 con 0x7fc134104c70 2023-07-14T19:24:05.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.854+0000 7fc1317fa700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] conn(0x7fc11c0aa9b0 0x7fc11c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.854+0000 7fc1317fa700 1 -- 172.21.15.53:0/696271819 --> [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc11c0ad470 con 0x7fc11c0aa9b0 2023-07-14T19:24:05.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.854+0000 7fc1317fa700 1 -- 172.21.15.53:0/696271819 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc12c0ce760 con 0x7fc134104c70 2023-07-14T19:24:05.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.854+0000 7fc1337fe700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] conn(0x7fc11c0aa9b0 0x7fc11c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.855+0000 7fc1337fe700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] conn(0x7fc11c0aa9b0 0x7fc11c0acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:05.858 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.856+0000 7fc133fff700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc11c0a11e0 0x7fc11c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.858 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.856+0000 7fc133fff700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc11c0a11e0 0x7fc11c0a3690 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc124009a10 tx=0x7fc124009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:05.858 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.857+0000 7fc1317fa700 1 -- 172.21.15.53:0/696271819 <== osd.5 v2:172.21.15.104:6808/4101602275 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc11c0ad470 con 0x7fc11c0aa9b0 2023-07-14T19:24:05.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.873+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 --> [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc13404e340 con 0x7fc11c0aa9b0 2023-07-14T19:24:05.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc1317fa700 1 -- 172.21.15.53:0/696271819 <== osd.5 v2:172.21.15.104:6808/4101602275 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc13404e340 con 0x7fc11c0aa9b0 2023-07-14T19:24:05.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 >> [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] conn(0x7fc11c0aa9b0 msgr2=0x7fc11c0acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] conn(0x7fc11c0aa9b0 0x7fc11c0acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc11c0a11e0 msgr2=0x7fc11c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc11c0a11e0 0x7fc11c0a3690 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc124009a10 tx=0x7fc124009190 comp rx=0 tx=0).stop 2023-07-14T19:24:05.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc134104c70 msgr2=0x7fc13410b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 --2- 172.21.15.53:0/696271819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc134104c70 0x7fc13410b810 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009fd0 tx=0x7fc12c00bce0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc138b46700 1 -- 172.21.15.53:0/696271819 reap_dead start 2023-07-14T19:24:05.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 shutdown_connections 2023-07-14T19:24:05.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 >> 172.21.15.53:0/696271819 conn(0x7fc134075280 msgr2=0x7fc1340fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:05.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 shutdown_connections 2023-07-14T19:24:05.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.874+0000 7fc13a5a9700 1 -- 172.21.15.53:0/696271819 wait complete. 2023-07-14T19:24:06.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 -- 172.21.15.53:0/968323602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f971c0ff7e0 msgr2=0x7f971c1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 --2- 172.21.15.53:0/968323602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f971c0ff7e0 0x7f971c1037b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f970c009990 tx=0x7f970c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 -- 172.21.15.53:0/968323602 shutdown_connections 2023-07-14T19:24:06.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 --2- 172.21.15.53:0/968323602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f971c103ee0 0x7f971c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 --2- 172.21.15.53:0/968323602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f971c0fee40 0x7f971c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 --2- 172.21.15.53:0/968323602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f971c0ff7e0 0x7f971c1037b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 -- 172.21.15.53:0/968323602 >> 172.21.15.53:0/968323602 conn(0x7f971c075ca0 msgr2=0x7f971c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:06.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 -- 172.21.15.53:0/968323602 shutdown_connections 2023-07-14T19:24:06.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 -- 172.21.15.53:0/968323602 wait complete. 2023-07-14T19:24:06.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 Processor -- start 2023-07-14T19:24:06.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.193+0000 7f9723a65700 1 -- start start 2023-07-14T19:24:06.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9723a65700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f971c0fee40 0x7f971c196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9723a65700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f971c0ff7e0 0x7f971c1971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9723a65700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f971c103ee0 0x7f971c1a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9723a65700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c197b00 con 0x7f971c0ff7e0 2023-07-14T19:24:06.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9723a65700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1a2270 con 0x7f971c103ee0 2023-07-14T19:24:06.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9723a65700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1a23e0 con 0x7f971c0fee40 2023-07-14T19:24:06.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f971c0ff7e0 0x7f971c1971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721000700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f971c0ff7e0 0x7f971c1971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50574/0 (socket says 172.21.15.53:50574) 2023-07-14T19:24:06.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721000700 1 -- 172.21.15.53:0/3698421906 learned_addr learned my addr 172.21.15.53:0/3698421906 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:06.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721801700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f971c0fee40 0x7f971c196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9722002700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f971c103ee0 0x7f971c1a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721000700 1 -- 172.21.15.53:0/3698421906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f971c0fee40 msgr2=0x7f971c196cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721000700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f971c0fee40 0x7f971c196cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721000700 1 -- 172.21.15.53:0/3698421906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f971c103ee0 msgr2=0x7f971c1a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721000700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f971c103ee0 0x7f971c1a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721000700 1 -- 172.21.15.53:0/3698421906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c0095f0 con 0x7f971c0ff7e0 2023-07-14T19:24:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.194+0000 7f9721000700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f971c0ff7e0 0x7f971c1971f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f970c0095c0 tx=0x7f970c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.195+0000 7f97127fc700 1 -- 172.21.15.53:0/3698421906 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c01d070 con 0x7f971c0ff7e0 2023-07-14T19:24:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.195+0000 7f97127fc700 1 -- 172.21.15.53:0/3698421906 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f970c003960 con 0x7f971c0ff7e0 2023-07-14T19:24:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.195+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c1a2660 con 0x7f971c0ff7e0 2023-07-14T19:24:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.195+0000 7f97127fc700 1 -- 172.21.15.53:0/3698421906 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c0054d0 con 0x7f971c0ff7e0 2023-07-14T19:24:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.195+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c1a2a70 con 0x7f971c0ff7e0 2023-07-14T19:24:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.196+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f971c04e340 con 0x7f971c0ff7e0 2023-07-14T19:24:06.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.197+0000 7f97127fc700 1 -- 172.21.15.53:0/3698421906 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f970c003ad0 con 0x7f971c0ff7e0 2023-07-14T19:24:06.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.198+0000 7f97127fc700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97080a1260 0x7f97080a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.198+0000 7f97127fc700 1 -- 172.21.15.53:0/3698421906 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f970c0cdc60 con 0x7f971c0ff7e0 2023-07-14T19:24:06.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.198+0000 7f9721801700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97080a1260 0x7f97080a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.198+0000 7f97127fc700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] conn(0x7f97080aaa30 0x7f97080ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.198+0000 7f97127fc700 1 -- 172.21.15.53:0/3698421906 --> [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97080ad4f0 con 0x7f97080aaa30 2023-07-14T19:24:06.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.198+0000 7f97127fc700 1 -- 172.21.15.53:0/3698421906 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f970c0ce010 con 0x7f971c0ff7e0 2023-07-14T19:24:06.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.198+0000 7f9721801700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97080a1260 0x7f97080a3710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9718005950 tx=0x7f971800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.199+0000 7f9722002700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] conn(0x7f97080aaa30 0x7f97080ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.199+0000 7f9722002700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] conn(0x7f97080aaa30 0x7f97080ace40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.200+0000 7f97127fc700 1 -- 172.21.15.53:0/3698421906 <== osd.11 v2:172.21.15.104:6824/3513976129 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f97080ad4f0 con 0x7f97080aaa30 2023-07-14T19:24:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.217+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 --> [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f971c10ee70 con 0x7f97080aaa30 2023-07-14T19:24:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f97127fc700 1 -- 172.21.15.53:0/3698421906 <== osd.11 v2:172.21.15.104:6824/3513976129 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9714009ba0 con 0x7f97080aaa30 2023-07-14T19:24:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 >> [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] conn(0x7f97080aaa30 msgr2=0x7f97080ace40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] conn(0x7f97080aaa30 0x7f97080ace40 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97080a1260 msgr2=0x7f97080a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f97080a1260 0x7f97080a3710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9718005950 tx=0x7f971800b410 comp rx=0 tx=0).stop 2023-07-14T19:24:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f971c0ff7e0 msgr2=0x7f971c1971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 --2- 172.21.15.53:0/3698421906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f971c0ff7e0 0x7f971c1971f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f970c0095c0 tx=0x7f970c00f690 comp rx=0 tx=0).stop 2023-07-14T19:24:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9722002700 1 -- 172.21.15.53:0/3698421906 reap_dead start 2023-07-14T19:24:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 shutdown_connections 2023-07-14T19:24:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 >> 172.21.15.53:0/3698421906 conn(0x7f971c075ca0 msgr2=0x7f971c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 shutdown_connections 2023-07-14T19:24:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.218+0000 7f9723a65700 1 -- 172.21.15.53:0/3698421906 wait complete. 2023-07-14T19:24:06.289 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.137+0000 7f574f117700 1 -- 172.21.15.53:0/3575145738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57481018c0 msgr2=0x7f5748109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.290 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.137+0000 7f574f117700 1 --2- 172.21.15.53:0/3575145738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57481018c0 0x7f5748109d30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5734009990 tx=0x7f5734009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.290 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.137+0000 7f574f117700 1 -- 172.21.15.53:0/3575145738 shutdown_connections 2023-07-14T19:24:06.290 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.137+0000 7f574f117700 1 --2- 172.21.15.53:0/3575145738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f574810a3a0 0x7f574810c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.290 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.137+0000 7f574f117700 1 --2- 172.21.15.53:0/3575145738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5748100f20 0x7f57481012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.290 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.137+0000 7f574f117700 1 --2- 172.21.15.53:0/3575145738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57481018c0 0x7f5748109d30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.290 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.137+0000 7f574f117700 1 -- 172.21.15.53:0/3575145738 >> 172.21.15.53:0/3575145738 conn(0x7f5748075c90 msgr2=0x7f5748076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:06.291 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.137+0000 7f574f117700 1 -- 172.21.15.53:0/3575145738 shutdown_connections 2023-07-14T19:24:06.291 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.138+0000 7f574f117700 1 -- 172.21.15.53:0/3575145738 wait complete. 2023-07-14T19:24:06.291 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.138+0000 7f574f117700 1 Processor -- start 2023-07-14T19:24:06.291 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.138+0000 7f574f117700 1 -- start start 2023-07-14T19:24:06.291 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.138+0000 7f574f117700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5748100f20 0x7f57480feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.291 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.138+0000 7f574f117700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481018c0 0x7f57480ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.291 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.138+0000 7f574f117700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f574810a3a0 0x7f57480ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.291 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.138+0000 7f574f117700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57481a4330 con 0x7f574810a3a0 2023-07-14T19:24:06.292 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.138+0000 7f574f117700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57481a44a0 con 0x7f57481018c0 2023-07-14T19:24:06.292 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.138+0000 7f574f117700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57481a4660 con 0x7f5748100f20 2023-07-14T19:24:06.292 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574d6b4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f574810a3a0 0x7f57480ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.292 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574d6b4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f574810a3a0 0x7f57480ff640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50530/0 (socket says 172.21.15.53:50530) 2023-07-14T19:24:06.292 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574d6b4700 1 -- 172.21.15.53:0/2027291787 learned_addr learned my addr 172.21.15.53:0/2027291787 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:06.292 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f5747fff700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481018c0 0x7f57480ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.292 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574ceb3700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5748100f20 0x7f57480feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574d6b4700 1 -- 172.21.15.53:0/2027291787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5748100f20 msgr2=0x7f57480feb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574d6b4700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5748100f20 0x7f57480feb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574d6b4700 1 -- 172.21.15.53:0/2027291787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481018c0 msgr2=0x7f57480ff0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574d6b4700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481018c0 0x7f57480ff0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574d6b4700 1 -- 172.21.15.53:0/2027291787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57340095f0 con 0x7f574810a3a0 2023-07-14T19:24:06.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f5747fff700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57481018c0 0x7f57480ff0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:06.293 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574d6b4700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f574810a3a0 0x7f57480ff640 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5740007ee0 tx=0x7f574000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f5745ffb700 1 -- 172.21.15.53:0/2027291787 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740017070 con 0x7f574810a3a0 2023-07-14T19:24:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57481a4940 con 0x7f574810a3a0 2023-07-14T19:24:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f5745ffb700 1 -- 172.21.15.53:0/2027291787 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f574000f040 con 0x7f574810a3a0 2023-07-14T19:24:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.139+0000 7f5745ffb700 1 -- 172.21.15.53:0/2027291787 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5740005580 con 0x7f574810a3a0 2023-07-14T19:24:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.140+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57481a4e40 con 0x7f574810a3a0 2023-07-14T19:24:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.140+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f574810ffd0 con 0x7f574810a3a0 2023-07-14T19:24:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.142+0000 7f5745ffb700 1 -- 172.21.15.53:0/2027291787 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f574000db20 con 0x7f574810a3a0 2023-07-14T19:24:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.142+0000 7f5745ffb700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57380a11e0 0x7f57380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.143+0000 7f5745ffb700 1 -- 172.21.15.53:0/2027291787 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f57400cc780 con 0x7f574810a3a0 2023-07-14T19:24:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.143+0000 7f574ceb3700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57380a11e0 0x7f57380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.143+0000 7f5745ffb700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] conn(0x7f57380aa9b0 0x7f57380acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.143+0000 7f5745ffb700 1 -- 172.21.15.53:0/2027291787 --> [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57380ad470 con 0x7f57380aa9b0 2023-07-14T19:24:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.143+0000 7f5745ffb700 1 -- 172.21.15.53:0/2027291787 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57400ccaf0 con 0x7f574810a3a0 2023-07-14T19:24:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.143+0000 7f5747fff700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] conn(0x7f57380aa9b0 0x7f57380acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.143+0000 7f574ceb3700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57380a11e0 0x7f57380a3690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f573c005950 tx=0x7f573c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.296 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.143+0000 7f5747fff700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] conn(0x7f57380aa9b0 0x7f57380acdc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.296 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.144+0000 7f5745ffb700 1 -- 172.21.15.53:0/2027291787 <== osd.0 v2:172.21.15.53:6802/243717916 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f57380ad470 con 0x7f57380aa9b0 2023-07-14T19:24:06.296 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.161+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 --> [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f574804e340 con 0x7f57380aa9b0 2023-07-14T19:24:06.296 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.161+0000 7f5745ffb700 1 -- 172.21.15.53:0/2027291787 <== osd.0 v2:172.21.15.53:6802/243717916 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f574804e340 con 0x7f57380aa9b0 2023-07-14T19:24:06.297 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] conn(0x7f57380aa9b0 msgr2=0x7f57380acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.297 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] conn(0x7f57380aa9b0 0x7f57380acdc0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.297 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57380a11e0 msgr2=0x7f57380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.297 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f57380a11e0 0x7f57380a3690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f573c005950 tx=0x7f573c00b410 comp rx=0 tx=0).stop 2023-07-14T19:24:06.297 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f574810a3a0 msgr2=0x7f57480ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.297 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 --2- 172.21.15.53:0/2027291787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f574810a3a0 0x7f57480ff640 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5740007ee0 tx=0x7f574000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.297 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574d6b4700 1 -- 172.21.15.53:0/2027291787 reap_dead start 2023-07-14T19:24:06.298 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 shutdown_connections 2023-07-14T19:24:06.298 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 >> 172.21.15.53:0/2027291787 conn(0x7f5748075c90 msgr2=0x7f57480fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:06.298 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 shutdown_connections 2023-07-14T19:24:06.298 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:05.162+0000 7f574f117700 1 -- 172.21.15.53:0/2027291787 wait complete. 2023-07-14T19:24:06.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.075+0000 7f5c1cff0700 1 -- 172.21.15.53:0/3339781172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c180ffff0 msgr2=0x7f5c181003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.075+0000 7f5c1cff0700 1 --2- 172.21.15.53:0/3339781172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c180ffff0 0x7f5c181003c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5c080099c0 tx=0x7f5c08009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.076+0000 7f5c1cff0700 1 -- 172.21.15.53:0/3339781172 shutdown_connections 2023-07-14T19:24:06.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.076+0000 7f5c1cff0700 1 --2- 172.21.15.53:0/3339781172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c18105090 0x7f5c1810cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.076+0000 7f5c1cff0700 1 --2- 172.21.15.53:0/3339781172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c180ffff0 0x7f5c181003c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.076+0000 7f5c1cff0700 1 --2- 172.21.15.53:0/3339781172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c18100990 0x7f5c18104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.076+0000 7f5c1cff0700 1 -- 172.21.15.53:0/3339781172 >> 172.21.15.53:0/3339781172 conn(0x7f5c18077d00 msgr2=0x7f5c18078100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:06.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.076+0000 7f5c1cff0700 1 -- 172.21.15.53:0/3339781172 shutdown_connections 2023-07-14T19:24:06.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.076+0000 7f5c1cff0700 1 -- 172.21.15.53:0/3339781172 wait complete. 2023-07-14T19:24:06.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.076+0000 7f5c1cff0700 1 Processor -- start 2023-07-14T19:24:06.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.076+0000 7f5c1cff0700 1 -- start start 2023-07-14T19:24:06.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1cff0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c180ffff0 0x7f5c18195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1cff0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c18100990 0x7f5c181962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1cff0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c18105090 0x7f5c1819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1cff0700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1819ad70 con 0x7f5c180ffff0 2023-07-14T19:24:06.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1cff0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1819aee0 con 0x7f5c18105090 2023-07-14T19:24:06.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1cff0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1819b0a0 con 0x7f5c18100990 2023-07-14T19:24:06.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c180ffff0 0x7f5c18195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c180ffff0 0x7f5c18195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50556/0 (socket says 172.21.15.53:50556) 2023-07-14T19:24:06.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1659c700 1 -- 172.21.15.53:0/2712593369 learned_addr learned my addr 172.21.15.53:0/2712593369 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:06.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c16d9d700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c18105090 0x7f5c1819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c15d9b700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c18100990 0x7f5c181962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1659c700 1 -- 172.21.15.53:0/2712593369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c18100990 msgr2=0x7f5c181962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1659c700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5c18100990 0x7f5c181962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1659c700 1 -- 172.21.15.53:0/2712593369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c18105090 msgr2=0x7f5c1819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1659c700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5c18105090 0x7f5c1819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1659c700 1 -- 172.21.15.53:0/2712593369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c080095f0 con 0x7f5c180ffff0 2023-07-14T19:24:06.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1659c700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c180ffff0 0x7f5c18195db0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800b580 tx=0x7f5c0800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c077fe700 1 -- 172.21.15.53:0/2712593369 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0801d070 con 0x7f5c180ffff0 2023-07-14T19:24:06.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c077fe700 1 -- 172.21.15.53:0/2712593369 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c08005030 con 0x7f5c180ffff0 2023-07-14T19:24:06.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.077+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1819b320 con 0x7f5c180ffff0 2023-07-14T19:24:06.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.078+0000 7f5c077fe700 1 -- 172.21.15.53:0/2712593369 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08004ad0 con 0x7f5c180ffff0 2023-07-14T19:24:06.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.078+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c181a2be0 con 0x7f5c180ffff0 2023-07-14T19:24:06.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.078+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5c1804e340 con 0x7f5c180ffff0 2023-07-14T19:24:06.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.080+0000 7f5c077fe700 1 -- 172.21.15.53:0/2712593369 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5c08005410 con 0x7f5c180ffff0 2023-07-14T19:24:06.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.080+0000 7f5c077fe700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5c000a1260 0x7f5c000a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.506 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.081+0000 7f5c15d9b700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5c000a1260 0x7f5c000a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.506 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.081+0000 7f5c077fe700 1 -- 172.21.15.53:0/2712593369 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5c080cdc70 con 0x7f5c180ffff0 2023-07-14T19:24:06.506 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.081+0000 7f5c077fe700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] conn(0x7f5c000aaa30 0x7f5c000ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.506 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.081+0000 7f5c077fe700 1 -- 172.21.15.53:0/2712593369 --> [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c000ad4f0 con 0x7f5c000aaa30 2023-07-14T19:24:06.506 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.081+0000 7f5c077fe700 1 -- 172.21.15.53:0/2712593369 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c080ce110 con 0x7f5c180ffff0 2023-07-14T19:24:06.506 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.081+0000 7f5c15d9b700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5c000a1260 0x7f5c000a3710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5c181973d0 tx=0x7f5c0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.506 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.082+0000 7f5c16d9d700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] conn(0x7f5c000aaa30 0x7f5c000ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.507 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.082+0000 7f5c16d9d700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] conn(0x7f5c000aaa30 0x7f5c000ace40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.507 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.083+0000 7f5c077fe700 1 -- 172.21.15.53:0/2712593369 <== osd.1 v2:172.21.15.70:6800/1213220726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5c000ad4f0 con 0x7f5c000aaa30 2023-07-14T19:24:06.507 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.101+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 --> [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5c1810ff40 con 0x7f5c000aaa30 2023-07-14T19:24:06.507 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c077fe700 1 -- 172.21.15.53:0/2712593369 <== osd.1 v2:172.21.15.70:6800/1213220726 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5c10010040 con 0x7f5c000aaa30 2023-07-14T19:24:06.507 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 >> [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] conn(0x7f5c000aaa30 msgr2=0x7f5c000ace40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] conn(0x7f5c000aaa30 0x7f5c000ace40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5c000a1260 msgr2=0x7f5c000a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5c000a1260 0x7f5c000a3710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5c181973d0 tx=0x7f5c0c009260 comp rx=0 tx=0).stop 2023-07-14T19:24:06.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c180ffff0 msgr2=0x7f5c18195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 --2- 172.21.15.53:0/2712593369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c180ffff0 0x7f5c18195db0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800b580 tx=0x7f5c0800f690 comp rx=0 tx=0).stop 2023-07-14T19:24:06.508 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c16d9d700 1 -- 172.21.15.53:0/2712593369 reap_dead start 2023-07-14T19:24:06.509 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 shutdown_connections 2023-07-14T19:24:06.509 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 >> 172.21.15.53:0/2712593369 conn(0x7f5c18077d00 msgr2=0x7f5c181039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:06.509 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 shutdown_connections 2023-07-14T19:24:06.509 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.102+0000 7f5c1cff0700 1 -- 172.21.15.53:0/2712593369 wait complete. 2023-07-14T19:24:06.726 INFO:teuthology.orchestra.run.smithi053.stdout:154618822697 2023-07-14T19:24:06.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-14T19:24:06.769 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.767+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/3913752259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a24100f20 msgr2=0x7f8a241012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.767+0000 7f8a2a1f4700 1 --2- 172.21.15.53:0/3913752259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a24100f20 0x7f8a241012f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8a14009990 tx=0x7f8a14009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.767+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/3913752259 shutdown_connections 2023-07-14T19:24:06.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.767+0000 7f8a2a1f4700 1 --2- 172.21.15.53:0/3913752259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a2410a270 0x7f8a2410c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.767+0000 7f8a2a1f4700 1 --2- 172.21.15.53:0/3913752259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a24100f20 0x7f8a241012f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.767+0000 7f8a2a1f4700 1 --2- 172.21.15.53:0/3913752259 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a24101830 0x7f8a24109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.767+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/3913752259 >> 172.21.15.53:0/3913752259 conn(0x7f8a24075ca0 msgr2=0x7f8a240760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:06.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.767+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/3913752259 shutdown_connections 2023-07-14T19:24:06.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.767+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/3913752259 wait complete. 2023-07-14T19:24:06.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.768+0000 7f8a2a1f4700 1 Processor -- start 2023-07-14T19:24:06.772 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.768+0000 7f8a2a1f4700 1 -- start start 2023-07-14T19:24:06.772 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a2a1f4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a24100f20 0x7f8a240feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.772 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a2a1f4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a24101830 0x7f8a240ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.772 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a2a1f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a2410a270 0x7f8a240ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.773 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a2a1f4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a241a4220 con 0x7f8a24101830 2023-07-14T19:24:06.773 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a2a1f4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a241a4390 con 0x7f8a24100f20 2023-07-14T19:24:06.773 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a2a1f4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a241a4550 con 0x7f8a2410a270 2023-07-14T19:24:06.773 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a22ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a24101830 0x7f8a240ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.773 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a22ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a24101830 0x7f8a240ff0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50590/0 (socket says 172.21.15.53:50590) 2023-07-14T19:24:06.773 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a22ffd700 1 -- 172.21.15.53:0/1258361573 learned_addr learned my addr 172.21.15.53:0/1258361573 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:06.773 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a23fff700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a2410a270 0x7f8a240ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a237fe700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a24100f20 0x7f8a240feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a22ffd700 1 -- 172.21.15.53:0/1258361573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a2410a270 msgr2=0x7f8a240ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a22ffd700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a2410a270 0x7f8a240ff640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a22ffd700 1 -- 172.21.15.53:0/1258361573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a24100f20 msgr2=0x7f8a240feb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a22ffd700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a24100f20 0x7f8a240feb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a22ffd700 1 -- 172.21.15.53:0/1258361573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a140095f0 con 0x7f8a24101830 2023-07-14T19:24:06.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a22ffd700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a24101830 0x7f8a240ff0c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800e9a0 tx=0x7f8a1800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.769+0000 7f8a20ff9700 1 -- 172.21.15.53:0/1258361573 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a18009790 con 0x7f8a24101830 2023-07-14T19:24:06.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.770+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a241a4830 con 0x7f8a24101830 2023-07-14T19:24:06.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.770+0000 7f8a20ff9700 1 -- 172.21.15.53:0/1258361573 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a18010430 con 0x7f8a24101830 2023-07-14T19:24:06.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.770+0000 7f8a20ff9700 1 -- 172.21.15.53:0/1258361573 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a18018680 con 0x7f8a24101830 2023-07-14T19:24:06.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.770+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a241a4d30 con 0x7f8a24101830 2023-07-14T19:24:06.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.770+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a2404e340 con 0x7f8a24101830 2023-07-14T19:24:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.772+0000 7f8a20ff9700 1 -- 172.21.15.53:0/1258361573 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8a180105a0 con 0x7f8a24101830 2023-07-14T19:24:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.772+0000 7f8a20ff9700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8a0c0a11e0 0x7f8a0c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.773+0000 7f8a20ff9700 1 -- 172.21.15.53:0/1258361573 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8a18014070 con 0x7f8a24101830 2023-07-14T19:24:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.773+0000 7f8a237fe700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8a0c0a11e0 0x7f8a0c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.773+0000 7f8a20ff9700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] conn(0x7f8a0c0aa9b0 0x7f8a0c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.773+0000 7f8a20ff9700 1 -- 172.21.15.53:0/1258361573 --> [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a0c0ad470 con 0x7f8a0c0aa9b0 2023-07-14T19:24:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.773+0000 7f8a20ff9700 1 -- 172.21.15.53:0/1258361573 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a180cd220 con 0x7f8a24101830 2023-07-14T19:24:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.773+0000 7f8a237fe700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8a0c0a11e0 0x7f8a0c0a3690 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400b2e0 tx=0x7f8a14005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.773+0000 7f8a23fff700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] conn(0x7f8a0c0aa9b0 0x7f8a0c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.773+0000 7f8a23fff700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] conn(0x7f8a0c0aa9b0 0x7f8a0c0acdc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.774+0000 7f8a20ff9700 1 -- 172.21.15.53:0/1258361573 <== osd.2 v2:172.21.15.104:6800/1379504827 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8a0c0ad470 con 0x7f8a0c0aa9b0 2023-07-14T19:24:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.790+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 --> [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a2410feb0 con 0x7f8a0c0aa9b0 2023-07-14T19:24:06.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a20ff9700 1 -- 172.21.15.53:0/1258361573 <== osd.2 v2:172.21.15.104:6800/1379504827 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8a1c00f040 con 0x7f8a0c0aa9b0 2023-07-14T19:24:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 >> [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] conn(0x7f8a0c0aa9b0 msgr2=0x7f8a0c0acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a2a1f4700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] conn(0x7f8a0c0aa9b0 0x7f8a0c0acdc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8a0c0a11e0 msgr2=0x7f8a0c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a2a1f4700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8a0c0a11e0 0x7f8a0c0a3690 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400b2e0 tx=0x7f8a14005fb0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a24101830 msgr2=0x7f8a240ff0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a2a1f4700 1 --2- 172.21.15.53:0/1258361573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a24101830 0x7f8a240ff0c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800e9a0 tx=0x7f8a1800ecb0 comp rx=0 tx=0).stop 2023-07-14T19:24:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a23fff700 1 -- 172.21.15.53:0/1258361573 reap_dead start 2023-07-14T19:24:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 shutdown_connections 2023-07-14T19:24:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 >> 172.21.15.53:0/1258361573 conn(0x7f8a24075ca0 msgr2=0x7f8a24101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.791+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 shutdown_connections 2023-07-14T19:24:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:06.792+0000 7f8a2a1f4700 1 -- 172.21.15.53:0/1258361573 wait complete. 2023-07-14T19:24:07.113 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.006+0000 7fb674d4d700 1 -- 172.21.15.53:0/4267254858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6701008d0 msgr2=0x7fb6701048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.113 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.006+0000 7fb674d4d700 1 --2- 172.21.15.53:0/4267254858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6701008d0 0x7fb6701048a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb6580099c0 tx=0x7fb658009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.114 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.006+0000 7fb674d4d700 1 -- 172.21.15.53:0/4267254858 shutdown_connections 2023-07-14T19:24:07.114 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.006+0000 7fb674d4d700 1 --2- 172.21.15.53:0/4267254858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb670104fd0 0x7fb67010cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.114 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.006+0000 7fb674d4d700 1 --2- 172.21.15.53:0/4267254858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6700fff30 0x7fb670100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.114 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.006+0000 7fb674d4d700 1 --2- 172.21.15.53:0/4267254858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6701008d0 0x7fb6701048a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.114 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.006+0000 7fb674d4d700 1 -- 172.21.15.53:0/4267254858 >> 172.21.15.53:0/4267254858 conn(0x7fb6700fb790 msgr2=0x7fb6700fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:07.115 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.006+0000 7fb674d4d700 1 -- 172.21.15.53:0/4267254858 shutdown_connections 2023-07-14T19:24:07.115 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.006+0000 7fb674d4d700 1 -- 172.21.15.53:0/4267254858 wait complete. 2023-07-14T19:24:07.115 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb674d4d700 1 Processor -- start 2023-07-14T19:24:07.115 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb674d4d700 1 -- start start 2023-07-14T19:24:07.116 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb674d4d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6700fff30 0x7fb670195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.116 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb674d4d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6701008d0 0x7fb6701962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.116 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb674d4d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb670104fd0 0x7fb67019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb674d4d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67019ad30 con 0x7fb670104fd0 2023-07-14T19:24:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb674d4d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67019aea0 con 0x7fb6700fff30 2023-07-14T19:24:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb674d4d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67019b060 con 0x7fb6701008d0 2023-07-14T19:24:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb66ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb670104fd0 0x7fb67019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb66ed9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb670104fd0 0x7fb67019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50608/0 (socket says 172.21.15.53:50608) 2023-07-14T19:24:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb66ed9d700 1 -- 172.21.15.53:0/1814601087 learned_addr learned my addr 172.21.15.53:0/1814601087 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb66dd9b700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6701008d0 0x7fb6701962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.007+0000 7fb66e59c700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6700fff30 0x7fb670195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb66ed9d700 1 -- 172.21.15.53:0/1814601087 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6701008d0 msgr2=0x7fb6701962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb66ed9d700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6701008d0 0x7fb6701962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb66ed9d700 1 -- 172.21.15.53:0/1814601087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6700fff30 msgr2=0x7fb670195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb66ed9d700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6700fff30 0x7fb670195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb66ed9d700 1 -- 172.21.15.53:0/1814601087 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580095f0 con 0x7fb670104fd0 2023-07-14T19:24:07.119 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb66e59c700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6700fff30 0x7fb670195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:07.119 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb66ed9d700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb670104fd0 0x7fb67019a720 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb66800ef30 tx=0x7fb66800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:07.119 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb6677fe700 1 -- 172.21.15.53:0/1814601087 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668018070 con 0x7fb670104fd0 2023-07-14T19:24:07.119 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb6677fe700 1 -- 172.21.15.53:0/1814601087 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb66800f040 con 0x7fb670104fd0 2023-07-14T19:24:07.119 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67019b2e0 con 0x7fb670104fd0 2023-07-14T19:24:07.119 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb6677fe700 1 -- 172.21.15.53:0/1814601087 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66800cdb0 con 0x7fb670104fd0 2023-07-14T19:24:07.119 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.008+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67019b590 con 0x7fb670104fd0 2023-07-14T19:24:07.120 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.009+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb67004e340 con 0x7fb670104fd0 2023-07-14T19:24:07.120 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.010+0000 7fb6677fe700 1 -- 172.21.15.53:0/1814601087 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb668003a00 con 0x7fb670104fd0 2023-07-14T19:24:07.120 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.011+0000 7fb6677fe700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb65c0a11e0 0x7fb65c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.120 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.011+0000 7fb66e59c700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb65c0a11e0 0x7fb65c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.121 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.011+0000 7fb6677fe700 1 -- 172.21.15.53:0/1814601087 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb6680cc400 con 0x7fb670104fd0 2023-07-14T19:24:07.121 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.011+0000 7fb6677fe700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] conn(0x7fb65c0aa9b0 0x7fb65c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.121 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.011+0000 7fb6677fe700 1 -- 172.21.15.53:0/1814601087 --> [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb65c0ad470 con 0x7fb65c0aa9b0 2023-07-14T19:24:07.121 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.011+0000 7fb6677fe700 1 -- 172.21.15.53:0/1814601087 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6680ce9e0 con 0x7fb670104fd0 2023-07-14T19:24:07.121 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.011+0000 7fb66e59c700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb65c0a11e0 0x7fb65c0a3690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb660009cb0 tx=0x7fb660009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:07.121 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.011+0000 7fb66dd9b700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] conn(0x7fb65c0aa9b0 0x7fb65c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.012+0000 7fb66dd9b700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] conn(0x7fb65c0aa9b0 0x7fb65c0acdc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.012+0000 7fb6677fe700 1 -- 172.21.15.53:0/1814601087 <== osd.4 v2:172.21.15.70:6808/195242658 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb65c0ad470 con 0x7fb65c0aa9b0 2023-07-14T19:24:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.031+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 --> [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb67010fe70 con 0x7fb65c0aa9b0 2023-07-14T19:24:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.031+0000 7fb6677fe700 1 -- 172.21.15.53:0/1814601087 <== osd.4 v2:172.21.15.70:6808/195242658 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb65801e040 con 0x7fb65c0aa9b0 2023-07-14T19:24:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 >> [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] conn(0x7fb65c0aa9b0 msgr2=0x7fb65c0acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] conn(0x7fb65c0aa9b0 0x7fb65c0acdc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb65c0a11e0 msgr2=0x7fb65c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb65c0a11e0 0x7fb65c0a3690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb660009cb0 tx=0x7fb660009260 comp rx=0 tx=0).stop 2023-07-14T19:24:07.123 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb670104fd0 msgr2=0x7fb67019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.123 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 --2- 172.21.15.53:0/1814601087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb670104fd0 0x7fb67019a720 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb66800ef30 tx=0x7fb66800c3a0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.123 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb66ed9d700 1 -- 172.21.15.53:0/1814601087 reap_dead start 2023-07-14T19:24:07.123 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 shutdown_connections 2023-07-14T19:24:07.123 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 >> 172.21.15.53:0/1814601087 conn(0x7fb6700fb790 msgr2=0x7fb6700fc260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:07.123 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 shutdown_connections 2023-07-14T19:24:07.123 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.032+0000 7fb674d4d700 1 -- 172.21.15.53:0/1814601087 wait complete. 2023-07-14T19:24:07.466 INFO:teuthology.orchestra.run.smithi053.stdout:240518168604 2023-07-14T19:24:07.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-07-14T19:24:07.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.441+0000 7f628ab7d700 1 -- 172.21.15.53:0/2650568961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62841095e0 msgr2=0x7f628410b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.441+0000 7f628ab7d700 1 --2- 172.21.15.53:0/2650568961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62841095e0 0x7f628410b9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b0a0 tx=0x7f627c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.441+0000 7f628ab7d700 1 -- 172.21.15.53:0/2650568961 shutdown_connections 2023-07-14T19:24:07.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.441+0000 7f628ab7d700 1 --2- 172.21.15.53:0/2650568961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62841095e0 0x7f628410b9c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.441+0000 7f628ab7d700 1 --2- 172.21.15.53:0/2650568961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6284100030 0x7f6284100400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.441+0000 7f628ab7d700 1 --2- 172.21.15.53:0/2650568961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6284100940 0x7f62841090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.441+0000 7f628ab7d700 1 -- 172.21.15.53:0/2650568961 >> 172.21.15.53:0/2650568961 conn(0x7f6284075480 msgr2=0x7f6284075880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:07.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.441+0000 7f628ab7d700 1 -- 172.21.15.53:0/2650568961 shutdown_connections 2023-07-14T19:24:07.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.442+0000 7f628ab7d700 1 -- 172.21.15.53:0/2650568961 wait complete. 2023-07-14T19:24:07.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.442+0000 7f628ab7d700 1 Processor -- start 2023-07-14T19:24:07.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.442+0000 7f628ab7d700 1 -- start start 2023-07-14T19:24:07.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.442+0000 7f628ab7d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6284100030 0x7f6284194b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.442+0000 7f628ab7d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6284100940 0x7f6284195070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.442+0000 7f628ab7d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62841095e0 0x7f62841955b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.442+0000 7f628ab7d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62841a4370 con 0x7f6284100940 2023-07-14T19:24:07.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.442+0000 7f628ab7d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62841a44e0 con 0x7f6284100030 2023-07-14T19:24:07.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.442+0000 7f628ab7d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62841a46a0 con 0x7f62841095e0 2023-07-14T19:24:07.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f6283fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6284100940 0x7f6284195070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f6283fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6284100940 0x7f6284195070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50622/0 (socket says 172.21.15.53:50622) 2023-07-14T19:24:07.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f6283fff700 1 -- 172.21.15.53:0/3242149466 learned_addr learned my addr 172.21.15.53:0/3242149466 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:07.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f628911a700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62841095e0 0x7f62841955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f6288919700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6284100030 0x7f6284194b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f628911a700 1 -- 172.21.15.53:0/3242149466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6284100030 msgr2=0x7f6284194b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f628911a700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6284100030 0x7f6284194b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f628911a700 1 -- 172.21.15.53:0/3242149466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6284100940 msgr2=0x7f6284195070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f628911a700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6284100940 0x7f6284195070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f628911a700 1 -- 172.21.15.53:0/3242149466 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c009d00 con 0x7f62841095e0 2023-07-14T19:24:07.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f6283fff700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6284100940 0x7f6284195070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:07.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.443+0000 7f628911a700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62841095e0 0x7f62841955b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6284072b10 tx=0x7f627c0079b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:07.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.444+0000 7f6281ffb700 1 -- 172.21.15.53:0/3242149466 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c00e040 con 0x7f62841095e0 2023-07-14T19:24:07.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.444+0000 7f6281ffb700 1 -- 172.21.15.53:0/3242149466 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f627c006080 con 0x7f62841095e0 2023-07-14T19:24:07.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.444+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62841a4920 con 0x7f62841095e0 2023-07-14T19:24:07.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.444+0000 7f6281ffb700 1 -- 172.21.15.53:0/3242149466 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c01b850 con 0x7f62841095e0 2023-07-14T19:24:07.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.444+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62841a4d90 con 0x7f62841095e0 2023-07-14T19:24:07.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.445+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f628404e340 con 0x7f62841095e0 2023-07-14T19:24:07.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.446+0000 7f6281ffb700 1 -- 172.21.15.53:0/3242149466 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f627c01d480 con 0x7f62841095e0 2023-07-14T19:24:07.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.447+0000 7f6281ffb700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f62740a11e0 0x7f62740a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.447+0000 7f6281ffb700 1 -- 172.21.15.53:0/3242149466 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f627c0cedb0 con 0x7f62841095e0 2023-07-14T19:24:07.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.447+0000 7f6288919700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f62740a11e0 0x7f62740a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.447+0000 7f6281ffb700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] conn(0x7f62740aa9b0 0x7f62740acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.447+0000 7f6281ffb700 1 -- 172.21.15.53:0/3242149466 --> [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62740ad470 con 0x7f62740aa9b0 2023-07-14T19:24:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.447+0000 7f6281ffb700 1 -- 172.21.15.53:0/3242149466 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f627c0cf250 con 0x7f62841095e0 2023-07-14T19:24:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.448+0000 7f6288919700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f62740a11e0 0x7f62740a3690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6278007be0 tx=0x7f62780073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.448+0000 7f6283fff700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] conn(0x7f62740aa9b0 0x7f62740acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.448+0000 7f6283fff700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] conn(0x7f62740aa9b0 0x7f62740acdc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.449+0000 7f6281ffb700 1 -- 172.21.15.53:0/3242149466 <== osd.7 v2:172.21.15.70:6816/2311839228 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f62740ad470 con 0x7f62740aa9b0 2023-07-14T19:24:07.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.470+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 --> [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f628410edd0 con 0x7f62740aa9b0 2023-07-14T19:24:07.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.470+0000 7f6281ffb700 1 -- 172.21.15.53:0/3242149466 <== osd.7 v2:172.21.15.70:6816/2311839228 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6270004020 con 0x7f62740aa9b0 2023-07-14T19:24:07.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 >> [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] conn(0x7f62740aa9b0 msgr2=0x7f62740acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] conn(0x7f62740aa9b0 0x7f62740acdc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f62740a11e0 msgr2=0x7f62740a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f62740a11e0 0x7f62740a3690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6278007be0 tx=0x7f62780073d0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62841095e0 msgr2=0x7f62841955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:07.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 --2- 172.21.15.53:0/3242149466 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62841095e0 0x7f62841955b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6284072b10 tx=0x7f627c0079b0 comp rx=0 tx=0).stop 2023-07-14T19:24:07.492 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628911a700 1 -- 172.21.15.53:0/3242149466 reap_dead start 2023-07-14T19:24:07.492 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 shutdown_connections 2023-07-14T19:24:07.492 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 >> 172.21.15.53:0/3242149466 conn(0x7f6284075480 msgr2=0x7f62840fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:07.492 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 shutdown_connections 2023-07-14T19:24:07.492 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:07.471+0000 7f628ab7d700 1 -- 172.21.15.53:0/3242149466 wait complete. 2023-07-14T19:24:07.629 INFO:teuthology.orchestra.run.smithi053.stdout:322122547217 2023-07-14T19:24:07.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-07-14T19:24:07.976 INFO:teuthology.orchestra.run.smithi053.stdout:133143986221 2023-07-14T19:24:07.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-14T19:24:08.076 INFO:teuthology.orchestra.run.smithi053.stdout:34359738431 2023-07-14T19:24:08.077 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-14T19:24:08.269 INFO:teuthology.orchestra.run.smithi053.stdout:184683593766 2023-07-14T19:24:08.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-14T19:24:08.506 INFO:teuthology.orchestra.run.smithi053.stdout:55834574906 2023-07-14T19:24:08.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-14T19:24:08.524 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:08 smithi053 ceph-mon[96565]: pgmap v263: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:24:08.525 INFO:teuthology.orchestra.run.smithi053.stdout:77309411382 2023-07-14T19:24:08.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-14T19:24:08.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:08 smithi070 ceph-mon[98015]: pgmap v263: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:24:08.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:08 smithi104 ceph-mon[98435]: pgmap v263: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:24:08.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.875+0000 7f88f8e84700 1 -- 172.21.15.53:0/924575065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88f4101830 msgr2=0x7f88f4109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:08.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.875+0000 7f88f8e84700 1 --2- 172.21.15.53:0/924575065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88f4101830 0x7f88f4109d30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009990 tx=0x7f88e8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:08.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.875+0000 7f88f8e84700 1 -- 172.21.15.53:0/924575065 shutdown_connections 2023-07-14T19:24:08.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.875+0000 7f88f8e84700 1 --2- 172.21.15.53:0/924575065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88f410a270 0x7f88f410c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:08.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.875+0000 7f88f8e84700 1 --2- 172.21.15.53:0/924575065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88f4100f20 0x7f88f41012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:08.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.875+0000 7f88f8e84700 1 --2- 172.21.15.53:0/924575065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88f4101830 0x7f88f4109d30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:08.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.875+0000 7f88f8e84700 1 -- 172.21.15.53:0/924575065 >> 172.21.15.53:0/924575065 conn(0x7f88f4075ca0 msgr2=0x7f88f40760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:08.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.875+0000 7f88f8e84700 1 -- 172.21.15.53:0/924575065 shutdown_connections 2023-07-14T19:24:08.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.875+0000 7f88f8e84700 1 -- 172.21.15.53:0/924575065 wait complete. 2023-07-14T19:24:08.887 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f8e84700 1 Processor -- start 2023-07-14T19:24:08.887 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f8e84700 1 -- start start 2023-07-14T19:24:08.887 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f8e84700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88f4100f20 0x7f88f4195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:08.888 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f8e84700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88f4101830 0x7f88f41962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:08.888 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f8e84700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88f410a270 0x7f88f419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:08.888 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f8e84700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f419ad70 con 0x7f88f4100f20 2023-07-14T19:24:08.889 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f8e84700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f419aee0 con 0x7f88f410a270 2023-07-14T19:24:08.889 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f8e84700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f419b0a0 con 0x7f88f4101830 2023-07-14T19:24:08.889 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88f4100f20 0x7f88f4195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:08.889 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88f4100f20 0x7f88f4195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50636/0 (socket says 172.21.15.53:50636) 2023-07-14T19:24:08.890 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f259c700 1 -- 172.21.15.53:0/2420672493 learned_addr learned my addr 172.21.15.53:0/2420672493 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:08.890 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f1d9b700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88f4101830 0x7f88f41962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:08.890 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f2d9d700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88f410a270 0x7f88f419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:08.891 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f259c700 1 -- 172.21.15.53:0/2420672493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88f4101830 msgr2=0x7f88f41962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:08.891 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f259c700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f88f4101830 0x7f88f41962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:08.891 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f259c700 1 -- 172.21.15.53:0/2420672493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88f410a270 msgr2=0x7f88f419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:08.891 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f259c700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88f410a270 0x7f88f419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:08.892 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.876+0000 7f88f259c700 1 -- 172.21.15.53:0/2420672493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e80095f0 con 0x7f88f4100f20 2023-07-14T19:24:08.892 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.877+0000 7f88f259c700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88f4100f20 0x7f88f4195db0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00c920 tx=0x7f88ec00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:08.892 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.877+0000 7f88e37fe700 1 -- 172.21.15.53:0/2420672493 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec007b00 con 0x7f88f4100f20 2023-07-14T19:24:08.892 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.877+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f419b380 con 0x7f88f4100f20 2023-07-14T19:24:08.893 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.877+0000 7f88e37fe700 1 -- 172.21.15.53:0/2420672493 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88ec00ce70 con 0x7f88f4100f20 2023-07-14T19:24:08.893 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.877+0000 7f88e37fe700 1 -- 172.21.15.53:0/2420672493 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec005460 con 0x7f88f4100f20 2023-07-14T19:24:08.893 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.877+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f41a2be0 con 0x7f88f4100f20 2023-07-14T19:24:08.894 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.878+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f88f410fed0 con 0x7f88f4100f20 2023-07-14T19:24:08.894 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.879+0000 7f88e37fe700 1 -- 172.21.15.53:0/2420672493 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f88ec004750 con 0x7f88f4100f20 2023-07-14T19:24:08.894 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.880+0000 7f88e37fe700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f88dc0a11e0 0x7f88dc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:08.894 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.880+0000 7f88f1d9b700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f88dc0a11e0 0x7f88dc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:08.895 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.880+0000 7f88e37fe700 1 -- 172.21.15.53:0/2420672493 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f88ec0cc9a0 con 0x7f88f4100f20 2023-07-14T19:24:08.895 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.880+0000 7f88e37fe700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] conn(0x7f88dc0aa9b0 0x7f88dc0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:08.895 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.880+0000 7f88e37fe700 1 -- 172.21.15.53:0/2420672493 --> [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88dc0ad470 con 0x7f88dc0aa9b0 2023-07-14T19:24:08.895 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.880+0000 7f88e37fe700 1 -- 172.21.15.53:0/2420672493 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88ec0cce40 con 0x7f88f4100f20 2023-07-14T19:24:08.896 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.880+0000 7f88f2d9d700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] conn(0x7f88dc0aa9b0 0x7f88dc0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:08.896 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.880+0000 7f88f1d9b700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f88dc0a11e0 0x7f88dc0a3690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f88f41973d0 tx=0x7f88e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:08.896 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.880+0000 7f88f2d9d700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] conn(0x7f88dc0aa9b0 0x7f88dc0acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:08.897 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.881+0000 7f88e37fe700 1 -- 172.21.15.53:0/2420672493 <== osd.9 v2:172.21.15.53:6826/3378335665 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f88dc0ad470 con 0x7f88dc0aa9b0 2023-07-14T19:24:08.901 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.900+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 --> [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f88f404e340 con 0x7f88dc0aa9b0 2023-07-14T19:24:08.902 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.900+0000 7f88e37fe700 1 -- 172.21.15.53:0/2420672493 <== osd.9 v2:172.21.15.53:6826/3378335665 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f88f404e340 con 0x7f88dc0aa9b0 2023-07-14T19:24:08.903 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] conn(0x7f88dc0aa9b0 msgr2=0x7f88dc0acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:08.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] conn(0x7f88dc0aa9b0 0x7f88dc0acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:08.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f88dc0a11e0 msgr2=0x7f88dc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:08.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f88dc0a11e0 0x7f88dc0a3690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f88f41973d0 tx=0x7f88e8005fd0 comp rx=0 tx=0).stop 2023-07-14T19:24:08.904 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88f4100f20 msgr2=0x7f88f4195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:08.905 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 --2- 172.21.15.53:0/2420672493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88f4100f20 0x7f88f4195db0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00c920 tx=0x7f88ec00cc30 comp rx=0 tx=0).stop 2023-07-14T19:24:08.905 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f2d9d700 1 -- 172.21.15.53:0/2420672493 reap_dead start 2023-07-14T19:24:08.905 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 shutdown_connections 2023-07-14T19:24:08.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 >> 172.21.15.53:0/2420672493 conn(0x7f88f4075ca0 msgr2=0x7f88f4101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:08.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 shutdown_connections 2023-07-14T19:24:08.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:08.901+0000 7f88f8e84700 1 -- 172.21.15.53:0/2420672493 wait complete. 2023-07-14T19:24:09.132 INFO:teuthology.orchestra.run.smithi053.stdout:206158430240 2023-07-14T19:24:09.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-14T19:24:09.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:09 smithi070 ceph-mon[98015]: pgmap v264: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:24:09.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:09 smithi104 ceph-mon[98435]: pgmap v264: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:24:09.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:09 smithi053 ceph-mon[96565]: pgmap v264: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:24:09.956 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:10.071 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:10.098 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:10.292 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:10.303 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:10.453 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:10.455 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:11.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:11 smithi053 ceph-mon[96565]: pgmap v265: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:24:11.698 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:11.711 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:11.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:11 smithi070 ceph-mon[98015]: pgmap v265: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:24:11.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:11 smithi104 ceph-mon[98435]: pgmap v265: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:24:12.167 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:13.560 INFO:teuthology.orchestra.run.smithi053.stdout:270582939673 2023-07-14T19:24:13.561 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-07-14T19:24:13.642 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:13 smithi053 ceph-mon[96565]: pgmap v266: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-07-14T19:24:13.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:13 smithi070 ceph-mon[98015]: pgmap v266: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-07-14T19:24:13.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:13 smithi104 ceph-mon[98435]: pgmap v266: 97 pgs: 97 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-07-14T19:24:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.909+0000 7f2f47d69700 1 -- 172.21.15.53:0/95749536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400fe8f0 msgr2=0x7f2f401038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.909+0000 7f2f47d69700 1 --2- 172.21.15.53:0/95749536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400fe8f0 0x7f2f401038d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800b0a0 tx=0x7f2f3800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.911+0000 7f2f47d69700 1 -- 172.21.15.53:0/95749536 shutdown_connections 2023-07-14T19:24:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.911+0000 7f2f47d69700 1 --2- 172.21.15.53:0/95749536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400fe8f0 0x7f2f401038d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.911+0000 7f2f47d69700 1 --2- 172.21.15.53:0/95749536 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40104bd0 0x7f2f40104fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.911+0000 7f2f47d69700 1 --2- 172.21.15.53:0/95749536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f401054e0 0x7f2f400fe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.911+0000 7f2f47d69700 1 -- 172.21.15.53:0/95749536 >> 172.21.15.53:0/95749536 conn(0x7f2f40075280 msgr2=0x7f2f40075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.912+0000 7f2f47d69700 1 -- 172.21.15.53:0/95749536 shutdown_connections 2023-07-14T19:24:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.912+0000 7f2f47d69700 1 -- 172.21.15.53:0/95749536 wait complete. 2023-07-14T19:24:13.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f47d69700 1 Processor -- start 2023-07-14T19:24:13.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f47d69700 1 -- start start 2023-07-14T19:24:13.914 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f47d69700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400fe8f0 0x7f2f4019c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:13.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f47d69700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40104bd0 0x7f2f4019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:13.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f47d69700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f401054e0 0x7f2f401a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:13.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f47d69700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f401a1430 con 0x7f2f401054e0 2023-07-14T19:24:13.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f47d69700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f401a15a0 con 0x7f2f400fe8f0 2023-07-14T19:24:13.915 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f47d69700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f401a1760 con 0x7f2f40104bd0 2023-07-14T19:24:13.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f45b05700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400fe8f0 0x7f2f4019c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:13.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f45b05700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400fe8f0 0x7f2f4019c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.53:37494/0 (socket says 172.21.15.53:37494) 2023-07-14T19:24:13.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f45b05700 1 -- 172.21.15.53:0/903726459 learned_addr learned my addr 172.21.15.53:0/903726459 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:13.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f46306700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f401054e0 0x7f2f401a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:13.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.913+0000 7f2f45304700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40104bd0 0x7f2f4019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:13.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.914+0000 7f2f45304700 1 -- 172.21.15.53:0/903726459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400fe8f0 msgr2=0x7f2f4019c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:13.916 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.914+0000 7f2f45304700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400fe8f0 0x7f2f4019c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.914+0000 7f2f45304700 1 -- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f401054e0 msgr2=0x7f2f401a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.914+0000 7f2f45304700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f401054e0 0x7f2f401a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.914+0000 7f2f45304700 1 -- 172.21.15.53:0/903726459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f38009d00 con 0x7f2f40104bd0 2023-07-14T19:24:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.914+0000 7f2f46306700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f401054e0 0x7f2f401a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.914+0000 7f2f45304700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40104bd0 0x7f2f4019c9b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f40105910 tx=0x7f2f3000b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.915+0000 7f2f36ffd700 1 -- 172.21.15.53:0/903726459 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30011830 con 0x7f2f40104bd0 2023-07-14T19:24:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.915+0000 7f2f36ffd700 1 -- 172.21.15.53:0/903726459 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f30011e50 con 0x7f2f40104bd0 2023-07-14T19:24:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.915+0000 7f2f36ffd700 1 -- 172.21.15.53:0/903726459 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30010dc0 con 0x7f2f40104bd0 2023-07-14T19:24:13.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.916+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f401a1a40 con 0x7f2f40104bd0 2023-07-14T19:24:13.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.916+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f401a92a0 con 0x7f2f40104bd0 2023-07-14T19:24:13.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.917+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4004e340 con 0x7f2f40104bd0 2023-07-14T19:24:13.921 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.920+0000 7f2f36ffd700 1 -- 172.21.15.53:0/903726459 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2f30010380 con 0x7f2f40104bd0 2023-07-14T19:24:13.922 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.921+0000 7f2f36ffd700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f2c0a1330 0x7f2f2c0a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:13.922 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.921+0000 7f2f36ffd700 1 -- 172.21.15.53:0/903726459 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2f300cc610 con 0x7f2f40104bd0 2023-07-14T19:24:13.922 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.921+0000 7f2f45b05700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f2c0a1330 0x7f2f2c0a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:13.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.922+0000 7f2f45b05700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f2c0a1330 0x7f2f2c0a37e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009cb0 tx=0x7f2f3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:13.924 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.923+0000 7f2f36ffd700 1 -- 172.21.15.53:0/903726459 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2f30095f40 con 0x7f2f40104bd0 2023-07-14T19:24:13.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.622+0000 7f6134060700 1 -- 172.21.15.53:0/141817286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 msgr2=0x7f612c1048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:13.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.622+0000 7f6134060700 1 --2- 172.21.15.53:0/141817286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 0x7f612c1048a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f611c0099c0 tx=0x7f611c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.623+0000 7f6134060700 1 -- 172.21.15.53:0/141817286 shutdown_connections 2023-07-14T19:24:13.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.623+0000 7f6134060700 1 --2- 172.21.15.53:0/141817286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c104fd0 0x7f612c10cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.623+0000 7f6134060700 1 --2- 172.21.15.53:0/141817286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f612c0fff30 0x7f612c100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.623+0000 7f6134060700 1 --2- 172.21.15.53:0/141817286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 0x7f612c1048a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.623+0000 7f6134060700 1 -- 172.21.15.53:0/141817286 >> 172.21.15.53:0/141817286 conn(0x7f612c0fb790 msgr2=0x7f612c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:13.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.623+0000 7f6134060700 1 -- 172.21.15.53:0/141817286 shutdown_connections 2023-07-14T19:24:13.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.623+0000 7f6134060700 1 -- 172.21.15.53:0/141817286 wait complete. 2023-07-14T19:24:13.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.623+0000 7f6134060700 1 Processor -- start 2023-07-14T19:24:13.963 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f6134060700 1 -- start start 2023-07-14T19:24:13.964 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f6134060700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f612c0fff30 0x7f612c195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:13.964 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f6134060700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 0x7f612c1963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:13.964 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f6134060700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c104fd0 0x7f612c19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:13.964 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f6134060700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c19ae60 con 0x7f612c1008d0 2023-07-14T19:24:13.964 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f6134060700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c19afd0 con 0x7f612c104fd0 2023-07-14T19:24:13.964 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f6134060700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c19b190 con 0x7f612c0fff30 2023-07-14T19:24:13.964 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f61315fb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 0x7f612c1963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:13.965 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f61315fb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 0x7f612c1963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50654/0 (socket says 172.21.15.53:50654) 2023-07-14T19:24:13.965 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f61315fb700 1 -- 172.21.15.53:0/1318505628 learned_addr learned my addr 172.21.15.53:0/1318505628 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:13.965 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f6131dfc700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f612c0fff30 0x7f612c195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:13.965 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f61325fd700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c104fd0 0x7f612c19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:13.965 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f61315fb700 1 -- 172.21.15.53:0/1318505628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f612c0fff30 msgr2=0x7f612c195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:13.965 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f61315fb700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f612c0fff30 0x7f612c195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.965 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f61315fb700 1 -- 172.21.15.53:0/1318505628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c104fd0 msgr2=0x7f612c19a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:13.966 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f61315fb700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c104fd0 0x7f612c19a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.966 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f61315fb700 1 -- 172.21.15.53:0/1318505628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c0095f0 con 0x7f612c1008d0 2023-07-14T19:24:13.966 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.624+0000 7f6131dfc700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f612c0fff30 0x7f612c195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:13.966 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.625+0000 7f61315fb700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 0x7f612c1963e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f611c009f50 tx=0x7f611c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:13.966 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.625+0000 7f6122ffd700 1 -- 172.21.15.53:0/1318505628 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c01d070 con 0x7f612c1008d0 2023-07-14T19:24:13.966 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.625+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c19b410 con 0x7f612c1008d0 2023-07-14T19:24:13.966 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.625+0000 7f6122ffd700 1 -- 172.21.15.53:0/1318505628 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f611c003f90 con 0x7f612c1008d0 2023-07-14T19:24:13.966 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.625+0000 7f6122ffd700 1 -- 172.21.15.53:0/1318505628 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c022bd0 con 0x7f612c1008d0 2023-07-14T19:24:13.967 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.625+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c072a40 con 0x7f612c1008d0 2023-07-14T19:24:13.967 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.626+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c04e340 con 0x7f612c1008d0 2023-07-14T19:24:13.967 INFO:teuthology.orchestra.run.smithi053.stdout:34359738433 2023-07-14T19:24:13.968 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.627+0000 7f6122ffd700 1 -- 172.21.15.53:0/1318505628 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f611c003ad0 con 0x7f612c1008d0 2023-07-14T19:24:13.968 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.628+0000 7f6122ffd700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f61180a11e0 0x7f61180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:13.968 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.628+0000 7f6131dfc700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f61180a11e0 0x7f61180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:13.968 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.628+0000 7f6122ffd700 1 -- 172.21.15.53:0/1318505628 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f611c0cdd20 con 0x7f612c1008d0 2023-07-14T19:24:13.968 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.629+0000 7f6131dfc700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f61180a11e0 0x7f61180a3690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6128007fd0 tx=0x7f61280073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:13.968 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.634+0000 7f6122ffd700 1 -- 172.21.15.53:0/1318505628 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f611c097550 con 0x7f612c1008d0 2023-07-14T19:24:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.886+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f612c072c60 con 0x7f612c1008d0 2023-07-14T19:24:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.886+0000 7f6122ffd700 1 -- 172.21.15.53:0/1318505628 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f611c09ab70 con 0x7f612c1008d0 2023-07-14T19:24:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.890+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f61180a11e0 msgr2=0x7f61180a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.890+0000 7f6134060700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f61180a11e0 0x7f61180a3690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6128007fd0 tx=0x7f61280073d0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.890+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 msgr2=0x7f612c1963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.890+0000 7f6134060700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 0x7f612c1963e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f611c009f50 tx=0x7f611c005740 comp rx=0 tx=0).stop 2023-07-14T19:24:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.893+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 shutdown_connections 2023-07-14T19:24:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.893+0000 7f6134060700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f61180a11e0 0x7f61180a3690 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.893+0000 7f6134060700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f612c104fd0 0x7f612c19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.893+0000 7f6134060700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f612c0fff30 0x7f612c195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.893+0000 7f6134060700 1 --2- 172.21.15.53:0/1318505628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f612c1008d0 0x7f612c1963e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:13.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.893+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 >> 172.21.15.53:0/1318505628 conn(0x7f612c0fb790 msgr2=0x7f612c0fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:13.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.893+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 shutdown_connections 2023-07-14T19:24:13.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.894+0000 7f6134060700 1 -- 172.21.15.53:0/1318505628 wait complete. 2023-07-14T19:24:14.170 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.169+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f2f4019d7d0 con 0x7f2f40104bd0 2023-07-14T19:24:14.171 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.170+0000 7f2f36ffd700 1 -- 172.21.15.53:0/903726459 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f2f30015070 con 0x7f2f40104bd0 2023-07-14T19:24:14.171 INFO:teuthology.orchestra.run.smithi053.stdout:322122547219 2023-07-14T19:24:14.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f2c0a1330 msgr2=0x7f2f2c0a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f2c0a1330 0x7f2f2c0a37e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009cb0 tx=0x7f2f3c009260 comp rx=0 tx=0).stop 2023-07-14T19:24:14.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40104bd0 msgr2=0x7f2f4019c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40104bd0 0x7f2f4019c9b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f40105910 tx=0x7f2f3000b730 comp rx=0 tx=0).stop 2023-07-14T19:24:14.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 shutdown_connections 2023-07-14T19:24:14.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2f2c0a1330 0x7f2f2c0a37e0 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009cb0 tx=0x7f2f3c009260 comp rx=0 tx=0).stop 2023-07-14T19:24:14.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f401054e0 0x7f2f401a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f400fe8f0 0x7f2f4019c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 --2- 172.21.15.53:0/903726459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f40104bd0 0x7f2f4019c9b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.172+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 >> 172.21.15.53:0/903726459 conn(0x7f2f40075280 msgr2=0x7f2f401018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:14.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.173+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 shutdown_connections 2023-07-14T19:24:14.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.173+0000 7f2f47d69700 1 -- 172.21.15.53:0/903726459 wait complete. 2023-07-14T19:24:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.298+0000 7f9f26820700 1 -- 172.21.15.53:0/595466888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f200fee70 msgr2=0x7f9f200ff240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.300 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.298+0000 7f9f26820700 1 --2- 172.21.15.53:0/595466888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f200fee70 0x7f9f200ff240 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9f100099c0 tx=0x7f9f10009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.300 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.298+0000 7f9f26820700 1 -- 172.21.15.53:0/595466888 shutdown_connections 2023-07-14T19:24:14.300 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.298+0000 7f9f26820700 1 --2- 172.21.15.53:0/595466888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f20103f10 0x7f9f2010ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.300 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.298+0000 7f9f26820700 1 --2- 172.21.15.53:0/595466888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f200fee70 0x7f9f200ff240 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.300 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.298+0000 7f9f26820700 1 --2- 172.21.15.53:0/595466888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f200ff810 0x7f9f201037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.298+0000 7f9f26820700 1 -- 172.21.15.53:0/595466888 >> 172.21.15.53:0/595466888 conn(0x7f9f20075d20 msgr2=0x7f9f20076120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.298+0000 7f9f26820700 1 -- 172.21.15.53:0/595466888 shutdown_connections 2023-07-14T19:24:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.298+0000 7f9f26820700 1 -- 172.21.15.53:0/595466888 wait complete. 2023-07-14T19:24:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.299+0000 7f9f26820700 1 Processor -- start 2023-07-14T19:24:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.299+0000 7f9f26820700 1 -- start start 2023-07-14T19:24:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.299+0000 7f9f26820700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f200fee70 0x7f9f20195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.299+0000 7f9f26820700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f200ff810 0x7f9f20196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.299+0000 7f9f26820700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f20103f10 0x7f9f2019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.299+0000 7f9f26820700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2019ad80 con 0x7f9f200fee70 2023-07-14T19:24:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.299+0000 7f9f26820700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2019aef0 con 0x7f9f20103f10 2023-07-14T19:24:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.299+0000 7f9f26820700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2019b0b0 con 0x7f9f200ff810 2023-07-14T19:24:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f200ff810 0x7f9f20196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f200ff810 0x7f9f20196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:50944/0 (socket says 172.21.15.53:50944) 2023-07-14T19:24:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f24dbd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f20103f10 0x7f9f2019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1f7fe700 1 -- 172.21.15.53:0/2545281919 learned_addr learned my addr 172.21.15.53:0/2545281919 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1f7fe700 1 -- 172.21.15.53:0/2545281919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f20103f10 msgr2=0x7f9f2019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1f7fe700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f20103f10 0x7f9f2019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1f7fe700 1 -- 172.21.15.53:0/2545281919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f200fee70 msgr2=0x7f9f20195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1f7fe700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f200fee70 0x7f9f20195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1f7fe700 1 -- 172.21.15.53:0/2545281919 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f08009520 con 0x7f9f200ff810 2023-07-14T19:24:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f24dbd700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f20103f10 0x7f9f2019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1f7fe700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f200ff810 0x7f9f20196300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800ea30 tx=0x7f9f0800ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.300+0000 7f9f1d7fa700 1 -- 172.21.15.53:0/2545281919 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08009820 con 0x7f9f200ff810 2023-07-14T19:24:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.301+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f100095f0 con 0x7f9f200ff810 2023-07-14T19:24:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.301+0000 7f9f1d7fa700 1 -- 172.21.15.53:0/2545281919 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f080044e0 con 0x7f9f200ff810 2023-07-14T19:24:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.301+0000 7f9f1d7fa700 1 -- 172.21.15.53:0/2545281919 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08003dc0 con 0x7f9f200ff810 2023-07-14T19:24:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.301+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f201a2a20 con 0x7f9f200ff810 2023-07-14T19:24:14.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.302+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2004e340 con 0x7f9f200ff810 2023-07-14T19:24:14.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.303+0000 7f9f1d7fa700 1 -- 172.21.15.53:0/2545281919 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9f08004000 con 0x7f9f200ff810 2023-07-14T19:24:14.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.303+0000 7f9f1d7fa700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9f0c0a11e0 0x7f9f0c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:14.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.304+0000 7f9f1ffff700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9f0c0a11e0 0x7f9f0c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:14.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.304+0000 7f9f1d7fa700 1 -- 172.21.15.53:0/2545281919 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9f08014070 con 0x7f9f200ff810 2023-07-14T19:24:14.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.304+0000 7f9f1ffff700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9f0c0a11e0 0x7f9f0c0a3690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009fd0 tx=0x7f9f10003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:14.308 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.307+0000 7f9f1d7fa700 1 -- 172.21.15.53:0/2545281919 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9f08096590 con 0x7f9f200ff810 2023-07-14T19:24:14.426 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:14 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1318505628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:24:14.427 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:14 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1194539299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:24:14.427 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:14 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/903726459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:24:14.523 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.521+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f9f20197120 con 0x7f9f200ff810 2023-07-14T19:24:14.524 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.522+0000 7f9f1d7fa700 1 -- 172.21.15.53:0/2545281919 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9f08094e60 con 0x7f9f200ff810 2023-07-14T19:24:14.524 INFO:teuthology.orchestra.run.smithi053.stdout:240518168606 2023-07-14T19:24:14.525 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9f0c0a11e0 msgr2=0x7f9f0c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9f0c0a11e0 0x7f9f0c0a3690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9f10009fd0 tx=0x7f9f10003680 comp rx=0 tx=0).stop 2023-07-14T19:24:14.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f200ff810 msgr2=0x7f9f20196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f200ff810 0x7f9f20196300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800ea30 tx=0x7f9f0800ed40 comp rx=0 tx=0).stop 2023-07-14T19:24:14.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 shutdown_connections 2023-07-14T19:24:14.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9f0c0a11e0 0x7f9f0c0a3690 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f20103f10 0x7f9f2019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.526 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f200fee70 0x7f9f20195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 --2- 172.21.15.53:0/2545281919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f200ff810 0x7f9f20196300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 >> 172.21.15.53:0/2545281919 conn(0x7f9f20075d20 msgr2=0x7f9f20068a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:14.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 shutdown_connections 2023-07-14T19:24:14.527 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.524+0000 7f9f26820700 1 -- 172.21.15.53:0/2545281919 wait complete. 2023-07-14T19:24:14.529 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.866+0000 7f0dc447b700 1 -- 172.21.15.53:0/1135466146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0ff7e0 msgr2=0x7f0dbc1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.866+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1135466146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0ff7e0 0x7f0dbc1037b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0db00099c0 tx=0x7f0db0009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.866+0000 7f0dc447b700 1 -- 172.21.15.53:0/1135466146 shutdown_connections 2023-07-14T19:24:14.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.866+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1135466146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dbc103ee0 0x7f0dbc10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.866+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1135466146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dbc0fee40 0x7f0dbc0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.866+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1135466146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0ff7e0 0x7f0dbc1037b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.866+0000 7f0dc447b700 1 -- 172.21.15.53:0/1135466146 >> 172.21.15.53:0/1135466146 conn(0x7f0dbc075ca0 msgr2=0x7f0dbc0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:14.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.867+0000 7f0dc447b700 1 -- 172.21.15.53:0/1135466146 shutdown_connections 2023-07-14T19:24:14.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.867+0000 7f0dc447b700 1 -- 172.21.15.53:0/1135466146 wait complete. 2023-07-14T19:24:14.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.867+0000 7f0dc447b700 1 Processor -- start 2023-07-14T19:24:14.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc447b700 1 -- start start 2023-07-14T19:24:14.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc447b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0fee40 0x7f0dbc195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:14.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc447b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dbc0ff7e0 0x7f0dbc196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:14.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc447b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dbc103ee0 0x7f0dbc19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:14.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc447b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc19ad70 con 0x7f0dbc0fee40 2023-07-14T19:24:14.531 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc447b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc19aee0 con 0x7f0dbc103ee0 2023-07-14T19:24:14.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc447b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc19b0a0 con 0x7f0dbc0ff7e0 2023-07-14T19:24:14.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc2217700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0fee40 0x7f0dbc195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:14.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc2217700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0fee40 0x7f0dbc195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50666/0 (socket says 172.21.15.53:50666) 2023-07-14T19:24:14.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc2217700 1 -- 172.21.15.53:0/1194539299 learned_addr learned my addr 172.21.15.53:0/1194539299 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:14.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc2a18700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dbc103ee0 0x7f0dbc19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:14.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc1a16700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dbc0ff7e0 0x7f0dbc196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:14.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc2217700 1 -- 172.21.15.53:0/1194539299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dbc0ff7e0 msgr2=0x7f0dbc196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc2217700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dbc0ff7e0 0x7f0dbc196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.868+0000 7f0dc2217700 1 -- 172.21.15.53:0/1194539299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dbc103ee0 msgr2=0x7f0dbc19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.869+0000 7f0dc2217700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dbc103ee0 0x7f0dbc19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.869+0000 7f0dc2217700 1 -- 172.21.15.53:0/1194539299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db00095f0 con 0x7f0dbc0fee40 2023-07-14T19:24:14.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.869+0000 7f0dc2217700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0fee40 0x7f0dbc195e00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0db800e9c0 tx=0x7f0db800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:14.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.869+0000 7f0daf7fe700 1 -- 172.21.15.53:0/1194539299 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db80097b0 con 0x7f0dbc0fee40 2023-07-14T19:24:14.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.869+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc19b380 con 0x7f0dbc0fee40 2023-07-14T19:24:14.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.869+0000 7f0daf7fe700 1 -- 172.21.15.53:0/1194539299 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0db800ee60 con 0x7f0dbc0fee40 2023-07-14T19:24:14.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.869+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc1a2be0 con 0x7f0dbc0fee40 2023-07-14T19:24:14.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.869+0000 7f0daf7fe700 1 -- 172.21.15.53:0/1194539299 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db80054f0 con 0x7f0dbc0fee40 2023-07-14T19:24:14.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.872+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da0005320 con 0x7f0dbc0fee40 2023-07-14T19:24:14.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.872+0000 7f0daf7fe700 1 -- 172.21.15.53:0/1194539299 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0db8004750 con 0x7f0dbc0fee40 2023-07-14T19:24:14.534 INFO:teuthology.orchestra.run.smithi053.stdout:154618822699 2023-07-14T19:24:14.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.873+0000 7f0daf7fe700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0da80a11e0 0x7f0da80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:14.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.873+0000 7f0daf7fe700 1 -- 172.21.15.53:0/1194539299 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0db8014070 con 0x7f0dbc0fee40 2023-07-14T19:24:14.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.874+0000 7f0dc1a16700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0da80a11e0 0x7f0da80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:14.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.877+0000 7f0dc1a16700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0da80a11e0 0x7f0da80a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009fd0 tx=0x7f0db0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:14.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:13.878+0000 7f0daf7fe700 1 -- 172.21.15.53:0/1194539299 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0db8096260 con 0x7f0dbc0fee40 2023-07-14T19:24:14.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.136+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0da0005cc0 con 0x7f0dbc0fee40 2023-07-14T19:24:14.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.136+0000 7f0daf7fe700 1 -- 172.21.15.53:0/1194539299 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0db8099880 con 0x7f0dbc0fee40 2023-07-14T19:24:14.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0da80a11e0 msgr2=0x7f0da80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0da80a11e0 0x7f0da80a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009fd0 tx=0x7f0db0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0fee40 msgr2=0x7f0dbc195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:14.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0fee40 0x7f0dbc195e00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0db800e9c0 tx=0x7f0db800ecd0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 shutdown_connections 2023-07-14T19:24:14.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0da80a11e0 0x7f0da80a3690 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dbc103ee0 0x7f0dbc19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dbc0fee40 0x7f0dbc195e00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 --2- 172.21.15.53:0/1194539299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0dbc0ff7e0 0x7f0dbc196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:14.538 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.138+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 >> 172.21.15.53:0/1194539299 conn(0x7f0dbc075ca0 msgr2=0x7f0dbc0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:14.538 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.139+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 shutdown_connections 2023-07-14T19:24:14.538 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:14.139+0000 7f0dc447b700 1 -- 172.21.15.53:0/1194539299 wait complete. 2023-07-14T19:24:14.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:14 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1318505628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:24:14.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:14 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1194539299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:24:14.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:14 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/903726459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:24:14.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:14 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1318505628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:24:14.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:14 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1194539299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:24:14.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:14 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/903726459' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:24:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:15 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2545281919' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:24:15.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:15 smithi053 ceph-mon[96565]: pgmap v267: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:24:15.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:15 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2545281919' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:24:15.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:15 smithi070 ceph-mon[98015]: pgmap v267: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:24:15.751 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547217 got 322122547219 for osd.11 2023-07-14T19:24:15.751 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:15.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:15 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2545281919' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:24:15.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:15 smithi104 ceph-mon[98435]: pgmap v267: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:24:16.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738431 got 34359738433 for osd.0 2023-07-14T19:24:16.136 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:16.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.511+0000 7ff649169700 1 -- 172.21.15.53:0/287125574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 msgr2=0x7ff64410ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.511+0000 7ff649169700 1 --2- 172.21.15.53:0/287125574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 0x7ff64410ba60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00b0d0 tx=0x7ff63c00b3e0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.511+0000 7ff649169700 1 -- 172.21.15.53:0/287125574 shutdown_connections 2023-07-14T19:24:16.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.511+0000 7ff649169700 1 --2- 172.21.15.53:0/287125574 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 0x7ff64410ba60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.511+0000 7ff649169700 1 --2- 172.21.15.53:0/287125574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6440fee40 0x7ff6440ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.511+0000 7ff649169700 1 --2- 172.21.15.53:0/287125574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440ff7e0 0x7ff6441037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.511+0000 7ff649169700 1 -- 172.21.15.53:0/287125574 >> 172.21.15.53:0/287125574 conn(0x7ff644075ca0 msgr2=0x7ff6440760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:16.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.511+0000 7ff649169700 1 -- 172.21.15.53:0/287125574 shutdown_connections 2023-07-14T19:24:16.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.511+0000 7ff649169700 1 -- 172.21.15.53:0/287125574 wait complete. 2023-07-14T19:24:16.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff649169700 1 Processor -- start 2023-07-14T19:24:16.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff649169700 1 -- start start 2023-07-14T19:24:16.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff649169700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440fee40 0x7ff644198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff649169700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6440ff7e0 0x7ff6441985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff649169700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 0x7ff64419ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff649169700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64419d040 con 0x7ff644103ee0 2023-07-14T19:24:16.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff649169700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64419d1b0 con 0x7ff6440fee40 2023-07-14T19:24:16.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff649169700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64419d370 con 0x7ff6440ff7e0 2023-07-14T19:24:16.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff64359e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 0x7ff64419ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff64359e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 0x7ff64419ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50712/0 (socket says 172.21.15.53:50712) 2023-07-14T19:24:16.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.512+0000 7ff64359e700 1 -- 172.21.15.53:0/637052394 learned_addr learned my addr 172.21.15.53:0/637052394 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:16.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff642d9d700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440fee40 0x7ff644198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff64259c700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6440ff7e0 0x7ff6441985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff64359e700 1 -- 172.21.15.53:0/637052394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6440ff7e0 msgr2=0x7ff6441985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff64359e700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6440ff7e0 0x7ff6441985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff64359e700 1 -- 172.21.15.53:0/637052394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440fee40 msgr2=0x7ff644198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff64359e700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440fee40 0x7ff644198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff64359e700 1 -- 172.21.15.53:0/637052394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c009d00 con 0x7ff644103ee0 2023-07-14T19:24:16.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff64359e700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 0x7ff64419ca30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff63c007870 tx=0x7ff63c0078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:16.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff63bfff700 1 -- 172.21.15.53:0/637052394 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c00e040 con 0x7ff644103ee0 2023-07-14T19:24:16.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff63bfff700 1 -- 172.21.15.53:0/637052394 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff63c005070 con 0x7ff644103ee0 2023-07-14T19:24:16.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64419d5f0 con 0x7ff644103ee0 2023-07-14T19:24:16.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff63bfff700 1 -- 172.21.15.53:0/637052394 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c01b6c0 con 0x7ff644103ee0 2023-07-14T19:24:16.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.513+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6441a4ce0 con 0x7ff644103ee0 2023-07-14T19:24:16.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.514+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64404e340 con 0x7ff644103ee0 2023-07-14T19:24:16.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.515+0000 7ff63bfff700 1 -- 172.21.15.53:0/637052394 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff63c0051e0 con 0x7ff644103ee0 2023-07-14T19:24:16.502 INFO:teuthology.orchestra.run.smithi053.stdout:103079215155 2023-07-14T19:24:16.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.516+0000 7ff63bfff700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6300a11e0 0x7ff6300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.516+0000 7ff642d9d700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6300a11e0 0x7ff6300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.516+0000 7ff63bfff700 1 -- 172.21.15.53:0/637052394 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff63c0cda20 con 0x7ff644103ee0 2023-07-14T19:24:16.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.516+0000 7ff642d9d700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6300a11e0 0x7ff6300a3690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff634005950 tx=0x7ff63400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:16.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.521+0000 7ff63bfff700 1 -- 172.21.15.53:0/637052394 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff63c097250 con 0x7ff644103ee0 2023-07-14T19:24:16.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.739+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff6441a4f10 con 0x7ff644103ee0 2023-07-14T19:24:16.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.739+0000 7ff63bfff700 1 -- 172.21.15.53:0/637052394 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff63c09a870 con 0x7ff644103ee0 2023-07-14T19:24:16.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6300a11e0 msgr2=0x7ff6300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6300a11e0 0x7ff6300a3690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff634005950 tx=0x7ff63400b410 comp rx=0 tx=0).stop 2023-07-14T19:24:16.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 msgr2=0x7ff64419ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 0x7ff64419ca30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff63c007870 tx=0x7ff63c0078a0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 shutdown_connections 2023-07-14T19:24:16.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff6300a11e0 0x7ff6300a3690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff644103ee0 0x7ff64419ca30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6440fee40 0x7ff644198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 --2- 172.21.15.53:0/637052394 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6440ff7e0 0x7ff6441985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 >> 172.21.15.53:0/637052394 conn(0x7ff644075ca0 msgr2=0x7ff644068990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:16.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 shutdown_connections 2023-07-14T19:24:16.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:15.743+0000 7ff649169700 1 -- 172.21.15.53:0/637052394 wait complete. 2023-07-14T19:24:16.541 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.540+0000 7f3522831700 1 -- 172.21.15.53:0/1180432122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c104fd0 msgr2=0x7f351c10cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.542 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.540+0000 7f3522831700 1 --2- 172.21.15.53:0/1180432122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c104fd0 0x7f351c10cb50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f351400b0d0 tx=0x7f351400b3e0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.542 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.540+0000 7f3522831700 1 -- 172.21.15.53:0/1180432122 shutdown_connections 2023-07-14T19:24:16.542 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.540+0000 7f3522831700 1 --2- 172.21.15.53:0/1180432122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c104fd0 0x7f351c10cb50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.542 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.540+0000 7f3522831700 1 --2- 172.21.15.53:0/1180432122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f351c0fff30 0x7f351c100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.542 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.540+0000 7f3522831700 1 --2- 172.21.15.53:0/1180432122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c1008d0 0x7f351c1048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.542 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.540+0000 7f3522831700 1 -- 172.21.15.53:0/1180432122 >> 172.21.15.53:0/1180432122 conn(0x7f351c0fb790 msgr2=0x7f351c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:16.543 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.540+0000 7f3522831700 1 -- 172.21.15.53:0/1180432122 shutdown_connections 2023-07-14T19:24:16.543 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.540+0000 7f3522831700 1 -- 172.21.15.53:0/1180432122 wait complete. 2023-07-14T19:24:16.544 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f3522831700 1 Processor -- start 2023-07-14T19:24:16.544 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f3522831700 1 -- start start 2023-07-14T19:24:16.545 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f3522831700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c0fff30 0x7f351c195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.545 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f3522831700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f351c1008d0 0x7f351c1963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.545 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f3522831700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c104fd0 0x7f351c19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.545 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f3522831700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c19ae20 con 0x7f351c0fff30 2023-07-14T19:24:16.545 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f3522831700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c19af90 con 0x7f351c104fd0 2023-07-14T19:24:16.546 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f3522831700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c19b150 con 0x7f351c1008d0 2023-07-14T19:24:16.546 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f351bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c0fff30 0x7f351c195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.546 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f351bfff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c0fff30 0x7f351c195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50722/0 (socket says 172.21.15.53:50722) 2023-07-14T19:24:16.546 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.541+0000 7f351bfff700 1 -- 172.21.15.53:0/2924799985 learned_addr learned my addr 172.21.15.53:0/2924799985 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:16.546 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f3520dce700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c104fd0 0x7f351c19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.546 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f351b7fe700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f351c1008d0 0x7f351c1963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.546 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f351bfff700 1 -- 172.21.15.53:0/2924799985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f351c1008d0 msgr2=0x7f351c1963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.546 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f351bfff700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f351c1008d0 0x7f351c1963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.547 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f351bfff700 1 -- 172.21.15.53:0/2924799985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c104fd0 msgr2=0x7f351c19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.547 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f351bfff700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c104fd0 0x7f351c19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.547 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f351bfff700 1 -- 172.21.15.53:0/2924799985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3514009d00 con 0x7f351c0fff30 2023-07-14T19:24:16.547 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f351bfff700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c0fff30 0x7f351c195e60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f350c00c920 tx=0x7f350c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:16.547 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f35197fa700 1 -- 172.21.15.53:0/2924799985 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c004a30 con 0x7f351c0fff30 2023-07-14T19:24:16.547 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f35197fa700 1 -- 172.21.15.53:0/2924799985 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f350c0054e0 con 0x7f351c0fff30 2023-07-14T19:24:16.547 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f35197fa700 1 -- 172.21.15.53:0/2924799985 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c007bd0 con 0x7f351c0fff30 2023-07-14T19:24:16.548 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c19b430 con 0x7f351c0fff30 2023-07-14T19:24:16.548 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.542+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c1a2c90 con 0x7f351c0fff30 2023-07-14T19:24:16.548 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.543+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c04e340 con 0x7f351c0fff30 2023-07-14T19:24:16.549 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.544+0000 7f35197fa700 1 -- 172.21.15.53:0/2924799985 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f350c007d50 con 0x7f351c0fff30 2023-07-14T19:24:16.549 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.545+0000 7f35197fa700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35040a11e0 0x7f35040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.549 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.545+0000 7f351b7fe700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35040a11e0 0x7f35040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.549 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.545+0000 7f35197fa700 1 -- 172.21.15.53:0/2924799985 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f350c0cc930 con 0x7f351c0fff30 2023-07-14T19:24:16.549 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.545+0000 7f351b7fe700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35040a11e0 0x7f35040a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f351c197480 tx=0x7f351000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:16.550 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.548+0000 7f35197fa700 1 -- 172.21.15.53:0/2924799985 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f350c095fd0 con 0x7f351c0fff30 2023-07-14T19:24:16.607 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:16 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/637052394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:24:16.608 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822697 got 154618822699 for osd.5 2023-07-14T19:24:16.609 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:16.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:16 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/637052394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:24:16.759 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.758+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f351c1971c0 con 0x7f351c0fff30 2023-07-14T19:24:16.760 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.758+0000 7f35197fa700 1 -- 172.21.15.53:0/2924799985 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f350c01d020 con 0x7f351c0fff30 2023-07-14T19:24:16.760 INFO:teuthology.orchestra.run.smithi053.stdout:133143986222 2023-07-14T19:24:16.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:16 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/637052394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:24:16.762 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35040a11e0 msgr2=0x7f35040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.762 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35040a11e0 0x7f35040a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f351c197480 tx=0x7f351000b410 comp rx=0 tx=0).stop 2023-07-14T19:24:16.762 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c0fff30 msgr2=0x7f351c195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.762 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c0fff30 0x7f351c195e60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f350c00c920 tx=0x7f350c00cc30 comp rx=0 tx=0).stop 2023-07-14T19:24:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 shutdown_connections 2023-07-14T19:24:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35040a11e0 0x7f35040a3690 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f351c104fd0 0x7f351c19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f351c0fff30 0x7f351c195e60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 --2- 172.21.15.53:0/2924799985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f351c1008d0 0x7f351c1963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 >> 172.21.15.53:0/2924799985 conn(0x7f351c0fb790 msgr2=0x7f351c0fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 shutdown_connections 2023-07-14T19:24:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.761+0000 7f3522831700 1 -- 172.21.15.53:0/2924799985 wait complete. 2023-07-14T19:24:16.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.833+0000 7fca02ac8700 1 -- 172.21.15.53:0/809802602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc0fe550 msgr2=0x7fc9fc0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.833+0000 7fca02ac8700 1 --2- 172.21.15.53:0/809802602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc0fe550 0x7fc9fc0fe920 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc9f00099c0 tx=0x7fc9f0009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.833+0000 7fca02ac8700 1 -- 172.21.15.53:0/809802602 shutdown_connections 2023-07-14T19:24:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.833+0000 7fca02ac8700 1 --2- 172.21.15.53:0/809802602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9fc105090 0x7fc9fc10cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.833+0000 7fca02ac8700 1 --2- 172.21.15.53:0/809802602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc0fe550 0x7fc9fc0fe920 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.833+0000 7fca02ac8700 1 --2- 172.21.15.53:0/809802602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9fc0feef0 0x7fc9fc104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.833+0000 7fca02ac8700 1 -- 172.21.15.53:0/809802602 >> 172.21.15.53:0/809802602 conn(0x7fc9fc0751e0 msgr2=0x7fc9fc0755e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:16.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.833+0000 7fca02ac8700 1 -- 172.21.15.53:0/809802602 shutdown_connections 2023-07-14T19:24:16.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.833+0000 7fca02ac8700 1 -- 172.21.15.53:0/809802602 wait complete. 2023-07-14T19:24:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca02ac8700 1 Processor -- start 2023-07-14T19:24:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca02ac8700 1 -- start start 2023-07-14T19:24:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca02ac8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9fc0fe550 0x7fc9fc0ff4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca02ac8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9fc0feef0 0x7fc9fc0ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca02ac8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc105090 0x7fc9fc0fff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca02ac8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1a4170 con 0x7fc9fc105090 2023-07-14T19:24:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca02ac8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1a42e0 con 0x7fc9fc0feef0 2023-07-14T19:24:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca02ac8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1a44a0 con 0x7fc9fc0fe550 2023-07-14T19:24:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca01065700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc105090 0x7fc9fc0fff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca01065700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc105090 0x7fc9fc0fff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50736/0 (socket says 172.21.15.53:50736) 2023-07-14T19:24:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.834+0000 7fca01065700 1 -- 172.21.15.53:0/1046631173 learned_addr learned my addr 172.21.15.53:0/1046631173 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fca00864700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9fc0fe550 0x7fc9fc0ff4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fc9fbfff700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9fc0feef0 0x7fc9fc0ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fca01065700 1 -- 172.21.15.53:0/1046631173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9fc0fe550 msgr2=0x7fc9fc0ff4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fca01065700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9fc0fe550 0x7fc9fc0ff4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fca01065700 1 -- 172.21.15.53:0/1046631173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9fc0feef0 msgr2=0x7fc9fc0ffa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fca01065700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9fc0feef0 0x7fc9fc0ffa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fca01065700 1 -- 172.21.15.53:0/1046631173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f00095f0 con 0x7fc9fc105090 2023-07-14T19:24:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fc9fbfff700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9fc0feef0 0x7fc9fc0ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fca00864700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9fc0fe550 0x7fc9fc0ff4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fca01065700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc105090 0x7fc9fc0fff90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400ef30 tx=0x7fc9f400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fc9f9ffb700 1 -- 172.21.15.53:0/1046631173 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4018070 con 0x7fc9fc105090 2023-07-14T19:24:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9fc1a4720 con 0x7fc9fc105090 2023-07-14T19:24:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fc9f9ffb700 1 -- 172.21.15.53:0/1046631173 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9f400f040 con 0x7fc9fc105090 2023-07-14T19:24:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.835+0000 7fc9f9ffb700 1 -- 172.21.15.53:0/1046631173 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f400cdb0 con 0x7fc9fc105090 2023-07-14T19:24:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.836+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9fc1a4c20 con 0x7fc9fc105090 2023-07-14T19:24:16.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.837+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9fc04e340 con 0x7fc9fc105090 2023-07-14T19:24:16.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.838+0000 7fc9f9ffb700 1 -- 172.21.15.53:0/1046631173 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc9f4003a00 con 0x7fc9fc105090 2023-07-14T19:24:16.841 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.838+0000 7fc9f9ffb700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc9ec0a11e0 0x7fc9ec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:16.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.838+0000 7fca00864700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc9ec0a11e0 0x7fc9ec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:16.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.838+0000 7fc9f9ffb700 1 -- 172.21.15.53:0/1046631173 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc9f40cc480 con 0x7fc9fc105090 2023-07-14T19:24:16.842 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.839+0000 7fca00864700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc9ec0a11e0 0x7fc9ec0a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009990 tx=0x7fc9f001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:16.844 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:16.843+0000 7fc9f9ffb700 1 -- 172.21.15.53:0/1046631173 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc9f4095cb0 con 0x7fc9fc105090 2023-07-14T19:24:16.887 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:17.089 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.087+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc9fc1a4e40 con 0x7fc9fc105090 2023-07-14T19:24:17.089 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.088+0000 7fc9f9ffb700 1 -- 172.21.15.53:0/1046631173 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc9f40992d0 con 0x7fc9fc105090 2023-07-14T19:24:17.090 INFO:teuthology.orchestra.run.smithi053.stdout:184683593768 2023-07-14T19:24:17.091 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc9ec0a11e0 msgr2=0x7fc9ec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc9ec0a11e0 0x7fc9ec0a3690 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009990 tx=0x7fc9f001a040 comp rx=0 tx=0).stop 2023-07-14T19:24:17.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc105090 msgr2=0x7fc9fc0fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc105090 0x7fc9fc0fff90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400ef30 tx=0x7fc9f400c3a0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 shutdown_connections 2023-07-14T19:24:17.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc9ec0a11e0 0x7fc9ec0a3690 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9fc105090 0x7fc9fc0fff90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.093 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc9fc0fe550 0x7fc9fc0ff4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.093 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 --2- 172.21.15.53:0/1046631173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9fc0feef0 0x7fc9fc0ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.093 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 >> 172.21.15.53:0/1046631173 conn(0x7fc9fc0751e0 msgr2=0x7fc9fc0fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:17.093 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 shutdown_connections 2023-07-14T19:24:17.093 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.090+0000 7fca02ac8700 1 -- 172.21.15.53:0/1046631173 wait complete. 2023-07-14T19:24:17.311 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.309+0000 7fd642811700 1 -- 172.21.15.53:0/2643989337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 msgr2=0x7fd63c10bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.311 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.309+0000 7fd642811700 1 --2- 172.21.15.53:0/2643989337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 0x7fd63c10bb40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd63400b0d0 tx=0x7fd63400b3e0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.311 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.309+0000 7fd642811700 1 -- 172.21.15.53:0/2643989337 shutdown_connections 2023-07-14T19:24:17.311 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.309+0000 7fd642811700 1 --2- 172.21.15.53:0/2643989337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 0x7fd63c10bb40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.312 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.309+0000 7fd642811700 1 --2- 172.21.15.53:0/2643989337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd63c0fef40 0x7fd63c0ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.312 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.309+0000 7fd642811700 1 --2- 172.21.15.53:0/2643989337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63c1007a0 0x7fd63c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.312 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.309+0000 7fd642811700 1 -- 172.21.15.53:0/2643989337 >> 172.21.15.53:0/2643989337 conn(0x7fd63c075c90 msgr2=0x7fd63c076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:17.312 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.309+0000 7fd642811700 1 -- 172.21.15.53:0/2643989337 shutdown_connections 2023-07-14T19:24:17.312 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.309+0000 7fd642811700 1 -- 172.21.15.53:0/2643989337 wait complete. 2023-07-14T19:24:17.312 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd642811700 1 Processor -- start 2023-07-14T19:24:17.313 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd642811700 1 -- start start 2023-07-14T19:24:17.314 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd642811700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63c0fef40 0x7fd63c195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:17.314 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd642811700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd63c1007a0 0x7fd63c196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:17.314 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd642811700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 0x7fd63c19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:17.314 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd642811700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c19ae90 con 0x7fd63c101150 2023-07-14T19:24:17.315 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd642811700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c19b000 con 0x7fd63c1007a0 2023-07-14T19:24:17.315 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd642811700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c19b1c0 con 0x7fd63c0fef40 2023-07-14T19:24:17.315 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd640dae700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 0x7fd63c19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:17.315 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd640dae700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 0x7fd63c19a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50752/0 (socket says 172.21.15.53:50752) 2023-07-14T19:24:17.315 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd640dae700 1 -- 172.21.15.53:0/1282694832 learned_addr learned my addr 172.21.15.53:0/1282694832 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:17.315 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd63b7fe700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd63c1007a0 0x7fd63c196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:17.316 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.310+0000 7fd63bfff700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63c0fef40 0x7fd63c195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:17.316 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd640dae700 1 -- 172.21.15.53:0/1282694832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63c0fef40 msgr2=0x7fd63c195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.316 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd640dae700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63c0fef40 0x7fd63c195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.316 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd640dae700 1 -- 172.21.15.53:0/1282694832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd63c1007a0 msgr2=0x7fd63c196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.316 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd640dae700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd63c1007a0 0x7fd63c196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.317 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd640dae700 1 -- 172.21.15.53:0/1282694832 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd634009d00 con 0x7fd63c101150 2023-07-14T19:24:17.317 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd63b7fe700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd63c1007a0 0x7fd63c196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:17.317 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd640dae700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 0x7fd63c19a880 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd634007870 tx=0x7fd6340078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:17.317 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd6397fa700 1 -- 172.21.15.53:0/1282694832 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63400e040 con 0x7fd63c101150 2023-07-14T19:24:17.318 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63c19b440 con 0x7fd63c101150 2023-07-14T19:24:17.318 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd6397fa700 1 -- 172.21.15.53:0/1282694832 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd634005070 con 0x7fd63c101150 2023-07-14T19:24:17.318 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd6397fa700 1 -- 172.21.15.53:0/1282694832 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63401b6c0 con 0x7fd63c101150 2023-07-14T19:24:17.318 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.311+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c1a2d00 con 0x7fd63c101150 2023-07-14T19:24:17.320 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.313+0000 7fd6397fa700 1 -- 172.21.15.53:0/1282694832 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd6340051e0 con 0x7fd63c101150 2023-07-14T19:24:17.320 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.314+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63c04e340 con 0x7fd63c101150 2023-07-14T19:24:17.321 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.314+0000 7fd6397fa700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6240a1260 0x7fd6240a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:17.321 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.314+0000 7fd6397fa700 1 -- 172.21.15.53:0/1282694832 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd6340cd960 con 0x7fd63c101150 2023-07-14T19:24:17.321 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.314+0000 7fd63bfff700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6240a1260 0x7fd6240a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:17.321 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.315+0000 7fd63bfff700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6240a1260 0x7fd6240a3710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009c60 tx=0x7fd62c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:17.322 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.319+0000 7fd6397fa700 1 -- 172.21.15.53:0/1282694832 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd634097190 con 0x7fd63c101150 2023-07-14T19:24:17.550 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.549+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd63c1a2e90 con 0x7fd63c101150 2023-07-14T19:24:17.550 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.549+0000 7fd6397fa700 1 -- 172.21.15.53:0/1282694832 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd63409a7b0 con 0x7fd63c101150 2023-07-14T19:24:17.551 INFO:teuthology.orchestra.run.smithi053.stdout:77309411384 2023-07-14T19:24:17.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6240a1260 msgr2=0x7fd6240a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6240a1260 0x7fd6240a3710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009c60 tx=0x7fd62c009260 comp rx=0 tx=0).stop 2023-07-14T19:24:17.553 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 msgr2=0x7fd63c19a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.554 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 0x7fd63c19a880 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd634007870 tx=0x7fd6340078a0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.554 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 shutdown_connections 2023-07-14T19:24:17.554 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd6240a1260 0x7fd6240a3710 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.554 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd63c101150 0x7fd63c19a880 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.555 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd63c0fef40 0x7fd63c195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.555 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 --2- 172.21.15.53:0/1282694832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd63c1007a0 0x7fd63c196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.555 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 >> 172.21.15.53:0/1282694832 conn(0x7fd63c075c90 msgr2=0x7fd63c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:17.555 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 shutdown_connections 2023-07-14T19:24:17.555 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.552+0000 7fd642811700 1 -- 172.21.15.53:0/1282694832 wait complete. 2023-07-14T19:24:17.680 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:17 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2924799985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:24:17.680 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:17 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1046631173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:24:17.680 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:17 smithi053 ceph-mon[96565]: pgmap v268: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T19:24:17.681 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215153 got 103079215155 for osd.3 2023-07-14T19:24:17.681 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:17.683 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.682+0000 7f8621584700 1 -- 172.21.15.53:0/1973438175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 msgr2=0x7f861c0ff240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.682+0000 7f8621584700 1 --2- 172.21.15.53:0/1973438175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 0x7f861c0ff240 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f860c00b0d0 tx=0x7f860c00b3e0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.682+0000 7f8621584700 1 -- 172.21.15.53:0/1973438175 shutdown_connections 2023-07-14T19:24:17.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.682+0000 7f8621584700 1 --2- 172.21.15.53:0/1973438175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861c103f10 0x7f861c10ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.682+0000 7f8621584700 1 --2- 172.21.15.53:0/1973438175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 0x7f861c0ff240 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.685 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.682+0000 7f8621584700 1 --2- 172.21.15.53:0/1973438175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f861c0ff810 0x7f861c1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.685 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.682+0000 7f8621584700 1 -- 172.21.15.53:0/1973438175 >> 172.21.15.53:0/1973438175 conn(0x7f861c075d20 msgr2=0x7f861c076120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:17.685 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.682+0000 7f8621584700 1 -- 172.21.15.53:0/1973438175 shutdown_connections 2023-07-14T19:24:17.685 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.682+0000 7f8621584700 1 -- 172.21.15.53:0/1973438175 wait complete. 2023-07-14T19:24:17.686 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.683+0000 7f8621584700 1 Processor -- start 2023-07-14T19:24:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.683+0000 7f8621584700 1 -- start start 2023-07-14T19:24:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f8621584700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 0x7f861c194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f8621584700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f861c0ff810 0x7f861c195340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f8621584700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861c103f10 0x7f861c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f8621584700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c199dc0 con 0x7f861c0fee70 2023-07-14T19:24:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f8621584700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c199f30 con 0x7f861c103f10 2023-07-14T19:24:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f8621584700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c19a0f0 con 0x7f861c0ff810 2023-07-14T19:24:17.688 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 0x7f861c194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:17.688 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861affd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 0x7f861c194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50762/0 (socket says 172.21.15.53:50762) 2023-07-14T19:24:17.688 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861affd700 1 -- 172.21.15.53:0/3739187413 learned_addr learned my addr 172.21.15.53:0/3739187413 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:17.688 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861b7fe700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861c103f10 0x7f861c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:17.688 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861a7fc700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f861c0ff810 0x7f861c195340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:17.688 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861affd700 1 -- 172.21.15.53:0/3739187413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f861c0ff810 msgr2=0x7f861c195340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.688 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861affd700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f861c0ff810 0x7f861c195340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.689 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861affd700 1 -- 172.21.15.53:0/3739187413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861c103f10 msgr2=0x7f861c1997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.689 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861affd700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861c103f10 0x7f861c1997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.689 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861affd700 1 -- 172.21.15.53:0/3739187413 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c009d00 con 0x7f861c0fee70 2023-07-14T19:24:17.689 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.684+0000 7f861affd700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 0x7f861c194e00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f860c0078a0 tx=0x7f860c0078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:17.689 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.685+0000 7f8613fff700 1 -- 172.21.15.53:0/3739187413 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c00e040 con 0x7f861c0fee70 2023-07-14T19:24:17.689 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.685+0000 7f8613fff700 1 -- 172.21.15.53:0/3739187413 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f860c01e070 con 0x7f861c0fee70 2023-07-14T19:24:17.689 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.685+0000 7f8613fff700 1 -- 172.21.15.53:0/3739187413 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c025880 con 0x7f861c0fee70 2023-07-14T19:24:17.689 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.685+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c19a370 con 0x7f861c0fee70 2023-07-14T19:24:17.690 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.685+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861c068bf0 con 0x7f861c0fee70 2023-07-14T19:24:17.690 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.686+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861c04e340 con 0x7f861c0fee70 2023-07-14T19:24:17.690 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.688+0000 7f8613fff700 1 -- 172.21.15.53:0/3739187413 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f860c0259e0 con 0x7f861c0fee70 2023-07-14T19:24:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.688+0000 7f8613fff700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86080a11e0 0x7f86080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.688+0000 7f861a7fc700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86080a11e0 0x7f86080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.688+0000 7f8613fff700 1 -- 172.21.15.53:0/3739187413 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f860c020070 con 0x7f861c0fee70 2023-07-14T19:24:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.689+0000 7f861a7fc700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86080a11e0 0x7f86080a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f861c196420 tx=0x7f8604009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:17.693 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.692+0000 7f8613fff700 1 -- 172.21.15.53:0/3739187413 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f860c05e780 con 0x7f861c0fee70 2023-07-14T19:24:17.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:17 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2924799985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:24:17.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:17 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1046631173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:24:17.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:17 smithi070 ceph-mon[98015]: pgmap v268: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T19:24:17.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:17 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2924799985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:24:17.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:17 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1046631173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:24:17.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:17 smithi104 ceph-mon[98435]: pgmap v268: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T19:24:17.903 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.902+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f861c19a560 con 0x7f861c0fee70 2023-07-14T19:24:17.903 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.902+0000 7f8613fff700 1 -- 172.21.15.53:0/3739187413 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f860c099610 con 0x7f861c0fee70 2023-07-14T19:24:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:292057776150 2023-07-14T19:24:17.905 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86080a11e0 msgr2=0x7f86080a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.905 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86080a11e0 0x7f86080a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f861c196420 tx=0x7f8604009260 comp rx=0 tx=0).stop 2023-07-14T19:24:17.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 msgr2=0x7f861c194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:17.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 0x7f861c194e00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f860c0078a0 tx=0x7f860c0078d0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 shutdown_connections 2023-07-14T19:24:17.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86080a11e0 0x7f86080a3690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.906 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861c103f10 0x7f861c1997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f861c0fee70 0x7f861c194e00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 --2- 172.21.15.53:0/3739187413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f861c0ff810 0x7f861c195340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:17.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 >> 172.21.15.53:0/3739187413 conn(0x7f861c075d20 msgr2=0x7f861c1027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:17.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 shutdown_connections 2023-07-14T19:24:17.907 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:17.904+0000 7f8621584700 1 -- 172.21.15.53:0/3739187413 wait complete. 2023-07-14T19:24:18.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.235+0000 7feda78ba700 1 -- 172.21.15.53:0/1324775938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 msgr2=0x7feda01037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:18.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.235+0000 7feda78ba700 1 --2- 172.21.15.53:0/1324775938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 0x7feda01037b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fed900099c0 tx=0x7fed90009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.235+0000 7feda78ba700 1 -- 172.21.15.53:0/1324775938 shutdown_connections 2023-07-14T19:24:18.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.235+0000 7feda78ba700 1 --2- 172.21.15.53:0/1324775938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda0103ee0 0x7feda010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.305 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.235+0000 7feda78ba700 1 --2- 172.21.15.53:0/1324775938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feda00fee40 0x7feda00ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.235+0000 7feda78ba700 1 --2- 172.21.15.53:0/1324775938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 0x7feda01037b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.235+0000 7feda78ba700 1 -- 172.21.15.53:0/1324775938 >> 172.21.15.53:0/1324775938 conn(0x7feda0075ca0 msgr2=0x7feda00760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:18.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.236+0000 7feda78ba700 1 -- 172.21.15.53:0/1324775938 shutdown_connections 2023-07-14T19:24:18.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.236+0000 7feda78ba700 1 -- 172.21.15.53:0/1324775938 wait complete. 2023-07-14T19:24:18.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.236+0000 7feda78ba700 1 Processor -- start 2023-07-14T19:24:18.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.236+0000 7feda78ba700 1 -- start start 2023-07-14T19:24:18.306 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.236+0000 7feda78ba700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feda00fee40 0x7feda0196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:18.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.236+0000 7feda78ba700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 0x7feda01970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:18.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.236+0000 7feda78ba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda0103ee0 0x7feda01a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:18.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda78ba700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda01979c0 con 0x7feda00ff7e0 2023-07-14T19:24:18.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda78ba700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda01a2130 con 0x7feda00fee40 2023-07-14T19:24:18.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda78ba700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda01a22a0 con 0x7feda0103ee0 2023-07-14T19:24:18.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda4e55700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 0x7feda01970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:18.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda4e55700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 0x7feda01970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50778/0 (socket says 172.21.15.53:50778) 2023-07-14T19:24:18.307 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda4e55700 1 -- 172.21.15.53:0/4155564330 learned_addr learned my addr 172.21.15.53:0/4155564330 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:18.308 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda5656700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feda00fee40 0x7feda0196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:18.308 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda5e57700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda0103ee0 0x7feda01a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:18.308 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda4e55700 1 -- 172.21.15.53:0/4155564330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda0103ee0 msgr2=0x7feda01a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:18.308 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda4e55700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda0103ee0 0x7feda01a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.308 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda4e55700 1 -- 172.21.15.53:0/4155564330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feda00fee40 msgr2=0x7feda0196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:18.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda4e55700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feda00fee40 0x7feda0196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda4e55700 1 -- 172.21.15.53:0/4155564330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed900095f0 con 0x7feda00ff7e0 2023-07-14T19:24:18.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda5e57700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda0103ee0 0x7feda01a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:18.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda4e55700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 0x7feda01970b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fed90009990 tx=0x7fed90005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:18.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7fed967fc700 1 -- 172.21.15.53:0/4155564330 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9001d070 con 0x7feda00ff7e0 2023-07-14T19:24:18.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7fed967fc700 1 -- 172.21.15.53:0/4155564330 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed90004090 con 0x7feda00ff7e0 2023-07-14T19:24:18.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7fed967fc700 1 -- 172.21.15.53:0/4155564330 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed90022d20 con 0x7feda00ff7e0 2023-07-14T19:24:18.309 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda01a2520 con 0x7feda00ff7e0 2023-07-14T19:24:18.310 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.237+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda01a29c0 con 0x7feda00ff7e0 2023-07-14T19:24:18.310 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.239+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda004e340 con 0x7feda00ff7e0 2023-07-14T19:24:18.310 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.240+0000 7fed967fc700 1 -- 172.21.15.53:0/4155564330 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fed90022470 con 0x7feda00ff7e0 2023-07-14T19:24:18.310 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.240+0000 7fed967fc700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fed8c0a1260 0x7fed8c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:18.310 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.241+0000 7feda5656700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fed8c0a1260 0x7fed8c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:18.310 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.241+0000 7fed967fc700 1 -- 172.21.15.53:0/4155564330 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fed900cda70 con 0x7feda00ff7e0 2023-07-14T19:24:18.310 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.241+0000 7feda5656700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fed8c0a1260 0x7fed8c0a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fed9c005ea0 tx=0x7fed9c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:18.311 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.245+0000 7fed967fc700 1 -- 172.21.15.53:0/4155564330 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fed9005c780 con 0x7feda00ff7e0 2023-07-14T19:24:18.451 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.450+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7feda01a2d00 con 0x7feda00ff7e0 2023-07-14T19:24:18.452 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.450+0000 7fed967fc700 1 -- 172.21.15.53:0/4155564330 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fed900cfe60 con 0x7feda00ff7e0 2023-07-14T19:24:18.452 INFO:teuthology.orchestra.run.smithi053.stdout:55834574908 2023-07-14T19:24:18.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.452+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fed8c0a1260 msgr2=0x7fed8c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:18.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.452+0000 7feda78ba700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fed8c0a1260 0x7fed8c0a3710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fed9c005ea0 tx=0x7fed9c005e30 comp rx=0 tx=0).stop 2023-07-14T19:24:18.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.452+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 msgr2=0x7feda01970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:18.454 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.452+0000 7feda78ba700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 0x7feda01970b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fed90009990 tx=0x7fed90005e80 comp rx=0 tx=0).stop 2023-07-14T19:24:18.455 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.453+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 shutdown_connections 2023-07-14T19:24:18.455 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.453+0000 7feda78ba700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fed8c0a1260 0x7fed8c0a3710 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.455 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.453+0000 7feda78ba700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feda0103ee0 0x7feda01a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.455 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.453+0000 7feda78ba700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feda00fee40 0x7feda0196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.455 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.453+0000 7feda78ba700 1 --2- 172.21.15.53:0/4155564330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feda00ff7e0 0x7feda01970b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:18.455 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.453+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 >> 172.21.15.53:0/4155564330 conn(0x7feda0075ca0 msgr2=0x7feda00fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:18.456 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.453+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 shutdown_connections 2023-07-14T19:24:18.456 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:18.453+0000 7feda78ba700 1 -- 172.21.15.53:0/4155564330 wait complete. 2023-07-14T19:24:18.489 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168604 got 240518168606 for osd.8 2023-07-14T19:24:18.490 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:18.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:18 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1282694832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:24:18.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:18 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3739187413' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:24:18.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:18 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1282694832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:24:18.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:18 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3739187413' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:24:18.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:18 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1282694832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:24:18.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:18 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3739187413' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:24:18.836 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593766 got 184683593768 for osd.6 2023-07-14T19:24:18.837 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:18.991 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986221 got 133143986222 for osd.4 2023-07-14T19:24:18.992 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:19.217 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411382 got 77309411384 for osd.2 2023-07-14T19:24:19.217 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:19.244 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776147 got 292057776150 for osd.10 2023-07-14T19:24:19.244 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:19.357 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:19.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:19 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/4155564330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:24:19.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:19 smithi053 ceph-mon[96565]: pgmap v269: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T19:24:19.668 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574906 got 55834574908 for osd.1 2023-07-14T19:24:19.669 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:19.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:19 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/4155564330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:24:19.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:19 smithi070 ceph-mon[98015]: pgmap v269: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T19:24:19.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:19 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/4155564330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:24:19.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:19 smithi104 ceph-mon[98435]: pgmap v269: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T19:24:20.203 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.201+0000 7fe869f37700 1 -- 172.21.15.53:0/1531249264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 msgr2=0x7fe86410ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:20.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.201+0000 7fe869f37700 1 --2- 172.21.15.53:0/1531249264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 0x7fe86410ba60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00b0d0 tx=0x7fe85c00b3e0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.201+0000 7fe869f37700 1 -- 172.21.15.53:0/1531249264 shutdown_connections 2023-07-14T19:24:20.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.201+0000 7fe869f37700 1 --2- 172.21.15.53:0/1531249264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 0x7fe86410ba60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.201+0000 7fe869f37700 1 --2- 172.21.15.53:0/1531249264 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8640fee40 0x7fe8640ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.202+0000 7fe869f37700 1 --2- 172.21.15.53:0/1531249264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8640ff7e0 0x7fe8641037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.202+0000 7fe869f37700 1 -- 172.21.15.53:0/1531249264 >> 172.21.15.53:0/1531249264 conn(0x7fe864075ca0 msgr2=0x7fe8640760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:20.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.202+0000 7fe869f37700 1 -- 172.21.15.53:0/1531249264 shutdown_connections 2023-07-14T19:24:20.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.202+0000 7fe869f37700 1 -- 172.21.15.53:0/1531249264 wait complete. 2023-07-14T19:24:20.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.202+0000 7fe869f37700 1 Processor -- start 2023-07-14T19:24:20.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe869f37700 1 -- start start 2023-07-14T19:24:20.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe869f37700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8640fee40 0x7fe864196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:20.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe869f37700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8640ff7e0 0x7fe864197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:20.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe869f37700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 0x7fe8641a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:20.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe869f37700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe864197ab0 con 0x7fe864103ee0 2023-07-14T19:24:20.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe869f37700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8641a21f0 con 0x7fe8640fee40 2023-07-14T19:24:20.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe869f37700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8641a23b0 con 0x7fe8640ff7e0 2023-07-14T19:24:20.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe863fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 0x7fe8641a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:20.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe8637fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8640fee40 0x7fe864196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:20.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe863fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 0x7fe8641a1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50792/0 (socket says 172.21.15.53:50792) 2023-07-14T19:24:20.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe863fff700 1 -- 172.21.15.53:0/1423262637 learned_addr learned my addr 172.21.15.53:0/1423262637 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:20.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe862ffd700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8640ff7e0 0x7fe864197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:20.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe863fff700 1 -- 172.21.15.53:0/1423262637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8640ff7e0 msgr2=0x7fe864197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:20.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe863fff700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8640ff7e0 0x7fe864197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe863fff700 1 -- 172.21.15.53:0/1423262637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8640fee40 msgr2=0x7fe864196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:20.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.203+0000 7fe863fff700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8640fee40 0x7fe864196c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.204+0000 7fe863fff700 1 -- 172.21.15.53:0/1423262637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c009d00 con 0x7fe864103ee0 2023-07-14T19:24:20.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.204+0000 7fe863fff700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 0x7fe8641a1cb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009fd0 tx=0x7fe85c007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:20.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.204+0000 7fe860ff9700 1 -- 172.21.15.53:0/1423262637 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c00e040 con 0x7fe864103ee0 2023-07-14T19:24:20.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.204+0000 7fe860ff9700 1 -- 172.21.15.53:0/1423262637 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe85c004e40 con 0x7fe864103ee0 2023-07-14T19:24:20.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.204+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8641a2630 con 0x7fe864103ee0 2023-07-14T19:24:20.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.204+0000 7fe860ff9700 1 -- 172.21.15.53:0/1423262637 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c01b8d0 con 0x7fe864103ee0 2023-07-14T19:24:20.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.204+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8641a2aa0 con 0x7fe864103ee0 2023-07-14T19:24:20.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.205+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86404e340 con 0x7fe864103ee0 2023-07-14T19:24:20.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.206+0000 7fe860ff9700 1 -- 172.21.15.53:0/1423262637 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe85c0056a0 con 0x7fe864103ee0 2023-07-14T19:24:20.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.207+0000 7fe860ff9700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe84c0a11e0 0x7fe84c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:20.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.207+0000 7fe8637fe700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe84c0a11e0 0x7fe84c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:20.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.207+0000 7fe860ff9700 1 -- 172.21.15.53:0/1423262637 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe85c0ce490 con 0x7fe864103ee0 2023-07-14T19:24:20.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.207+0000 7fe8637fe700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe84c0a11e0 0x7fe84c0a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe85400ac50 tx=0x7fe85400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:20.212 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.211+0000 7fe860ff9700 1 -- 172.21.15.53:0/1423262637 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe85c097cc0 con 0x7fe864103ee0 2023-07-14T19:24:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.411+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fe8641a2df0 con 0x7fe864103ee0 2023-07-14T19:24:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.411+0000 7fe860ff9700 1 -- 172.21.15.53:0/1423262637 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe85c09b2e0 con 0x7fe864103ee0 2023-07-14T19:24:20.413 INFO:teuthology.orchestra.run.smithi053.stdout:206158430243 2023-07-14T19:24:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe84c0a11e0 msgr2=0x7fe84c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe84c0a11e0 0x7fe84c0a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe85400ac50 tx=0x7fe85400a380 comp rx=0 tx=0).stop 2023-07-14T19:24:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 msgr2=0x7fe8641a1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 0x7fe8641a1cb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009fd0 tx=0x7fe85c007870 comp rx=0 tx=0).stop 2023-07-14T19:24:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 shutdown_connections 2023-07-14T19:24:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe84c0a11e0 0x7fe84c0a3690 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe864103ee0 0x7fe8641a1cb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8640fee40 0x7fe864196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 --2- 172.21.15.53:0/1423262637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8640ff7e0 0x7fe864197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.413+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 >> 172.21.15.53:0/1423262637 conn(0x7fe864075ca0 msgr2=0x7fe8640fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.414+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 shutdown_connections 2023-07-14T19:24:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:20.414+0000 7fe869f37700 1 -- 172.21.15.53:0/1423262637 wait complete. 2023-07-14T19:24:20.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:20 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1423262637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:24:20.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:20 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1423262637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:24:20.764 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:20 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1423262637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:24:20.765 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430240 got 206158430243 for osd.7 2023-07-14T19:24:20.765 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:21.165 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.163+0000 7f85994a2700 1 -- 172.21.15.53:0/3733626097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85940fee90 msgr2=0x7f85940ff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:21.165 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.163+0000 7f85994a2700 1 --2- 172.21.15.53:0/3733626097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85940fee90 0x7f85940ff260 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f857c0099c0 tx=0x7f857c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.165 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.163+0000 7f85994a2700 1 -- 172.21.15.53:0/3733626097 shutdown_connections 2023-07-14T19:24:21.165 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.163+0000 7f85994a2700 1 --2- 172.21.15.53:0/3733626097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8594103f30 0x7f859410bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.165 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.163+0000 7f85994a2700 1 --2- 172.21.15.53:0/3733626097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85940fee90 0x7f85940ff260 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.166 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.163+0000 7f85994a2700 1 --2- 172.21.15.53:0/3733626097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85940ff830 0x7f8594103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.166 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.163+0000 7f85994a2700 1 -- 172.21.15.53:0/3733626097 >> 172.21.15.53:0/3733626097 conn(0x7f8594075ae0 msgr2=0x7f8594075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:21.166 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.163+0000 7f85994a2700 1 -- 172.21.15.53:0/3733626097 shutdown_connections 2023-07-14T19:24:21.166 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.163+0000 7f85994a2700 1 -- 172.21.15.53:0/3733626097 wait complete. 2023-07-14T19:24:21.167 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.164+0000 7f85994a2700 1 Processor -- start 2023-07-14T19:24:21.168 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.164+0000 7f85994a2700 1 -- start start 2023-07-14T19:24:21.168 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.164+0000 7f85994a2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85940fee90 0x7f8594195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:21.168 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.164+0000 7f85994a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85940ff830 0x7f85941963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:21.168 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.164+0000 7f85994a2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8594103f30 0x7f859419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:21.168 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.164+0000 7f85994a2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859419ad90 con 0x7f8594103f30 2023-07-14T19:24:21.168 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.164+0000 7f85994a2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859419af00 con 0x7f85940ff830 2023-07-14T19:24:21.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.164+0000 7f85994a2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859419b0c0 con 0x7f85940fee90 2023-07-14T19:24:21.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85937fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8594103f30 0x7f859419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:21.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85937fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8594103f30 0x7f859419a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50806/0 (socket says 172.21.15.53:50806) 2023-07-14T19:24:21.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85937fe700 1 -- 172.21.15.53:0/3104912286 learned_addr learned my addr 172.21.15.53:0/3104912286 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:21.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85927fc700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85940ff830 0x7f85941963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:21.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f8592ffd700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85940fee90 0x7f8594195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:21.169 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85937fe700 1 -- 172.21.15.53:0/3104912286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85940fee90 msgr2=0x7f8594195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:21.170 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85937fe700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85940fee90 0x7f8594195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.170 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85937fe700 1 -- 172.21.15.53:0/3104912286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85940ff830 msgr2=0x7f85941963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:21.170 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85937fe700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85940ff830 0x7f85941963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.170 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85937fe700 1 -- 172.21.15.53:0/3104912286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857c0095f0 con 0x7f8594103f30 2023-07-14T19:24:21.170 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85937fe700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8594103f30 0x7f859419a780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f858c00ef30 tx=0x7f858c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:21.171 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f858bfff700 1 -- 172.21.15.53:0/3104912286 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c018070 con 0x7f8594103f30 2023-07-14T19:24:21.172 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f858bfff700 1 -- 172.21.15.53:0/3104912286 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f858c00f040 con 0x7f8594103f30 2023-07-14T19:24:21.172 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859419b340 con 0x7f8594103f30 2023-07-14T19:24:21.172 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f858bfff700 1 -- 172.21.15.53:0/3104912286 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c00cd00 con 0x7f8594103f30 2023-07-14T19:24:21.172 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.165+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85941a2c00 con 0x7f8594103f30 2023-07-14T19:24:21.172 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.166+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859404e340 con 0x7f8594103f30 2023-07-14T19:24:21.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.168+0000 7f858bfff700 1 -- 172.21.15.53:0/3104912286 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f858c003a00 con 0x7f8594103f30 2023-07-14T19:24:21.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.168+0000 7f858bfff700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f85800a11e0 0x7f85800a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:21.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.168+0000 7f8592ffd700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f85800a11e0 0x7f85800a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:21.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.168+0000 7f858bfff700 1 -- 172.21.15.53:0/3104912286 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f858c0ccd60 con 0x7f8594103f30 2023-07-14T19:24:21.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.169+0000 7f8592ffd700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f85800a11e0 0x7f85800a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f857c009990 tx=0x7f857c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:21.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.171+0000 7f858bfff700 1 -- 172.21.15.53:0/3104912286 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f858c096590 con 0x7f8594103f30 2023-07-14T19:24:21.369 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.368+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f85941a2f20 con 0x7f8594103f30 2023-07-14T19:24:21.369 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.368+0000 7f858bfff700 1 -- 172.21.15.53:0/3104912286 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f858c094e60 con 0x7f8594103f30 2023-07-14T19:24:21.370 INFO:teuthology.orchestra.run.smithi053.stdout:270582939675 2023-07-14T19:24:21.371 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f85800a11e0 msgr2=0x7f85800a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:21.371 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f85800a11e0 0x7f85800a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f857c009990 tx=0x7f857c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.371 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8594103f30 msgr2=0x7f859419a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:21.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8594103f30 0x7f859419a780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f858c00ef30 tx=0x7f858c00c3a0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 shutdown_connections 2023-07-14T19:24:21.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f85800a11e0 0x7f85800a3690 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8594103f30 0x7f859419a780 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f85940fee90 0x7f8594195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 --2- 172.21.15.53:0/3104912286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85940ff830 0x7f85941963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 >> 172.21.15.53:0/3104912286 conn(0x7f8594075ae0 msgr2=0x7f85941029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:21.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 shutdown_connections 2023-07-14T19:24:21.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:21.370+0000 7f85994a2700 1 -- 172.21.15.53:0/3104912286 wait complete. 2023-07-14T19:24:21.712 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:21 smithi053 ceph-mon[96565]: pgmap v270: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:24:21.713 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:21 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3104912286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:24:21.726 INFO:tasks.cephadm.ceph_manager.ceph:need seq 270582939673 got 270582939675 for osd.9 2023-07-14T19:24:21.727 DEBUG:teuthology.parallel:result is None 2023-07-14T19:24:21.727 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-14T19:24:21.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:24:21.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:21 smithi070 ceph-mon[98015]: pgmap v270: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:24:21.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:21 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3104912286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:24:21.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:21 smithi104 ceph-mon[98435]: pgmap v270: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:24:21.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:21 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3104912286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:24:22.288 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:23.579 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.576+0000 7f991bc24700 1 -- 172.21.15.53:0/1856990260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 msgr2=0x7f99141012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:23.579 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.576+0000 7f991bc24700 1 --2- 172.21.15.53:0/1856990260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 0x7f99141012f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9910009990 tx=0x7f9910009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.576+0000 7f991bc24700 1 -- 172.21.15.53:0/1856990260 shutdown_connections 2023-07-14T19:24:23.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.576+0000 7f991bc24700 1 --2- 172.21.15.53:0/1856990260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f991410a270 0x7f991410c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.576+0000 7f991bc24700 1 --2- 172.21.15.53:0/1856990260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 0x7f99141012f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.576+0000 7f991bc24700 1 --2- 172.21.15.53:0/1856990260 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9914101830 0x7f9914109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.576+0000 7f991bc24700 1 -- 172.21.15.53:0/1856990260 >> 172.21.15.53:0/1856990260 conn(0x7f9914075ca0 msgr2=0x7f99140760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:23.580 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.576+0000 7f991bc24700 1 -- 172.21.15.53:0/1856990260 shutdown_connections 2023-07-14T19:24:23.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.576+0000 7f991bc24700 1 -- 172.21.15.53:0/1856990260 wait complete. 2023-07-14T19:24:23.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f991bc24700 1 Processor -- start 2023-07-14T19:24:23.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f991bc24700 1 -- start start 2023-07-14T19:24:23.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f991bc24700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 0x7f9914195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:23.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f991bc24700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9914101830 0x7f9914196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:23.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f991bc24700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f991410a270 0x7f991419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:23.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f991bc24700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991419ada0 con 0x7f9914100f20 2023-07-14T19:24:23.581 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f991bc24700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991419af10 con 0x7f991410a270 2023-07-14T19:24:23.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f991bc24700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991419b0d0 con 0x7f9914101830 2023-07-14T19:24:23.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f99199c0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 0x7f9914195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:23.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f99199c0700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 0x7f9914195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50814/0 (socket says 172.21.15.53:50814) 2023-07-14T19:24:23.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.577+0000 7f99199c0700 1 -- 172.21.15.53:0/2500283660 learned_addr learned my addr 172.21.15.53:0/2500283660 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:23.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f99191bf700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9914101830 0x7f9914196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:23.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f99199c0700 1 -- 172.21.15.53:0/2500283660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9914101830 msgr2=0x7f9914196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:23.582 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f991a1c1700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f991410a270 0x7f991419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:23.583 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f99199c0700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9914101830 0x7f9914196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.583 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f99199c0700 1 -- 172.21.15.53:0/2500283660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f991410a270 msgr2=0x7f991419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:23.583 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f99199c0700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f991410a270 0x7f991419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.583 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f99199c0700 1 -- 172.21.15.53:0/2500283660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99100095f0 con 0x7f9914100f20 2023-07-14T19:24:23.584 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f99199c0700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 0x7f9914195de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f99100095c0 tx=0x7f991000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:23.584 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f990affd700 1 -- 172.21.15.53:0/2500283660 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991001d070 con 0x7f9914100f20 2023-07-14T19:24:23.584 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991419b350 con 0x7f9914100f20 2023-07-14T19:24:23.584 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f990affd700 1 -- 172.21.15.53:0/2500283660 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f991000fc00 con 0x7f9914100f20 2023-07-14T19:24:23.584 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f990affd700 1 -- 172.21.15.53:0/2500283660 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910017650 con 0x7f9914100f20 2023-07-14T19:24:23.584 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.578+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99141a2c10 con 0x7f9914100f20 2023-07-14T19:24:23.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.579+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9914104ac0 con 0x7f9914100f20 2023-07-14T19:24:23.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.580+0000 7f990affd700 1 -- 172.21.15.53:0/2500283660 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9910022470 con 0x7f9914100f20 2023-07-14T19:24:23.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.581+0000 7f990affd700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99000a11e0 0x7f99000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:23.585 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.581+0000 7f99191bf700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99000a11e0 0x7f99000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:23.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.581+0000 7f990affd700 1 -- 172.21.15.53:0/2500283660 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f99100ce170 con 0x7f9914100f20 2023-07-14T19:24:23.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.581+0000 7f99191bf700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99000a11e0 0x7f99000a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9904005950 tx=0x7f99040058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:23.586 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.584+0000 7f990affd700 1 -- 172.21.15.53:0/2500283660 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f99100979a0 con 0x7f9914100f20 2023-07-14T19:24:23.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:23 smithi070 ceph-mon[98015]: pgmap v271: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T19:24:23.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:23 smithi104 ceph-mon[98435]: pgmap v271: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T19:24:23.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.777+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9914197280 con 0x7f99000a11e0 2023-07-14T19:24:23.791 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.789+0000 7f990affd700 1 -- 172.21.15.53:0/2500283660 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+339941 (secure 0 0 0) 0x7f990400b8f0 con 0x7f99000a11e0 2023-07-14T19:24:23.794 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:23.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99000a11e0 msgr2=0x7f99000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:23.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99000a11e0 0x7f99000a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9904005950 tx=0x7f99040058e0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 msgr2=0x7f9914195de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:23.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 0x7f9914195de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f99100095c0 tx=0x7f991000f710 comp rx=0 tx=0).stop 2023-07-14T19:24:23.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 shutdown_connections 2023-07-14T19:24:23.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99000a11e0 0x7f99000a3690 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f991410a270 0x7f991419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9914100f20 0x7f9914195de0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 --2- 172.21.15.53:0/2500283660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9914101830 0x7f9914196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:23.798 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 >> 172.21.15.53:0/2500283660 conn(0x7f9914075ca0 msgr2=0x7f9914101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:23.798 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 shutdown_connections 2023-07-14T19:24:23.798 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:23.793+0000 7f991bc24700 1 -- 172.21.15.53:0/2500283660 wait complete. 2023-07-14T19:24:23.798 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2023-07-14T19:24:23.817 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:23 smithi053 ceph-mon[96565]: pgmap v271: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2023-07-14T19:24:24.245 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2023-07-14T19:24:23.326032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653045,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":392,"num_read_kb":310,"num_write":332,"num_write_kb":3042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270,"ondisk_log_size":270,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336428,"kb_used_data":12236,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605396,"statfs":{"total":1151940427776,"available":1151595925504,"internally_reserved":0,"allocated":12529664,"data_stored":6920793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":32,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003072"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472803+0000","last_change":"2023-07-14T19:23:38.301785+0000","last_active":"2023-07-14T19:23:39.472803+0000","last_peered":"2023-07-14T19:23:39.472803+0000","last_clean":"2023-07-14T19:23:39.472803+0000","last_became_active":"2023-07-14T19:23:38.301014+0000","last_became_peered":"2023-07-14T19:23:38.301014+0000","last_unstale":"2023-07-14T19:23:39.472803+0000","last_undegraded":"2023-07-14T19:23:39.472803+0000","last_fullsized":"2023-07-14T19:23:39.472803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:02.663606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514628+0000","last_change":"2023-07-14T19:23:38.299101+0000","last_active":"2023-07-14T19:23:39.514628+0000","last_peered":"2023-07-14T19:23:39.514628+0000","last_clean":"2023-07-14T19:23:39.514628+0000","last_became_active":"2023-07-14T19:23:38.298720+0000","last_became_peered":"2023-07-14T19:23:38.298720+0000","last_unstale":"2023-07-14T19:23:39.514628+0000","last_undegraded":"2023-07-14T19:23:39.514628+0000","last_fullsized":"2023-07-14T19:23:39.514628+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:05:31.873298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514167+0000","last_change":"2023-07-14T19:23:38.295335+0000","last_active":"2023-07-14T19:23:39.514167+0000","last_peered":"2023-07-14T19:23:39.514167+0000","last_clean":"2023-07-14T19:23:39.514167+0000","last_became_active":"2023-07-14T19:23:38.295031+0000","last_became_peered":"2023-07-14T19:23:38.295031+0000","last_unstale":"2023-07-14T19:23:39.514167+0000","last_undegraded":"2023-07-14T19:23:39.514167+0000","last_fullsized":"2023-07-14T19:23:39.514167+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:09.443353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515703+0000","last_change":"2023-07-14T19:23:38.295055+0000","last_active":"2023-07-14T19:23:39.515703+0000","last_peered":"2023-07-14T19:23:39.515703+0000","last_clean":"2023-07-14T19:23:39.515703+0000","last_became_active":"2023-07-14T19:23:38.294751+0000","last_became_peered":"2023-07-14T19:23:38.294751+0000","last_unstale":"2023-07-14T19:23:39.515703+0000","last_undegraded":"2023-07-14T19:23:39.515703+0000","last_fullsized":"2023-07-14T19:23:39.515703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:19.020190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286436+0000","last_change":"2023-07-14T19:23:38.294684+0000","last_active":"2023-07-14T19:23:39.286436+0000","last_peered":"2023-07-14T19:23:39.286436+0000","last_clean":"2023-07-14T19:23:39.286436+0000","last_became_active":"2023-07-14T19:23:38.294368+0000","last_became_peered":"2023-07-14T19:23:38.294368+0000","last_unstale":"2023-07-14T19:23:39.286436+0000","last_undegraded":"2023-07-14T19:23:39.286436+0000","last_fullsized":"2023-07-14T19:23:39.286436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:35.017660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283944+0000","last_change":"2023-07-14T19:23:38.301219+0000","last_active":"2023-07-14T19:23:39.283944+0000","last_peered":"2023-07-14T19:23:39.283944+0000","last_clean":"2023-07-14T19:23:39.283944+0000","last_became_active":"2023-07-14T19:23:38.300904+0000","last_became_peered":"2023-07-14T19:23:38.300904+0000","last_unstale":"2023-07-14T19:23:39.283944+0000","last_undegraded":"2023-07-14T19:23:39.283944+0000","last_fullsized":"2023-07-14T19:23:39.283944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:53:02.603728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515586+0000","last_change":"2023-07-14T19:23:38.292831+0000","last_active":"2023-07-14T19:23:39.515586+0000","last_peered":"2023-07-14T19:23:39.515586+0000","last_clean":"2023-07-14T19:23:39.515586+0000","last_became_active":"2023-07-14T19:23:38.292536+0000","last_became_peered":"2023-07-14T19:23:38.292536+0000","last_unstale":"2023-07-14T19:23:39.515586+0000","last_undegraded":"2023-07-14T19:23:39.515586+0000","last_fullsized":"2023-07-14T19:23:39.515586+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:49.316491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283393+0000","last_change":"2023-07-14T19:23:38.304940+0000","last_active":"2023-07-14T19:23:39.283393+0000","last_peered":"2023-07-14T19:23:39.283393+0000","last_clean":"2023-07-14T19:23:39.283393+0000","last_became_active":"2023-07-14T19:23:38.304609+0000","last_became_peered":"2023-07-14T19:23:38.304609+0000","last_unstale":"2023-07-14T19:23:39.283393+0000","last_undegraded":"2023-07-14T19:23:39.283393+0000","last_fullsized":"2023-07-14T19:23:39.283393+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:14.673980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515328+0000","last_change":"2023-07-14T19:23:38.300273+0000","last_active":"2023-07-14T19:23:39.515328+0000","last_peered":"2023-07-14T19:23:39.515328+0000","last_clean":"2023-07-14T19:23:39.515328+0000","last_became_active":"2023-07-14T19:23:38.299965+0000","last_became_peered":"2023-07-14T19:23:38.299965+0000","last_unstale":"2023-07-14T19:23:39.515328+0000","last_undegraded":"2023-07-14T19:23:39.515328+0000","last_fullsized":"2023-07-14T19:23:39.515328+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:26:21.737406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284243+0000","last_change":"2023-07-14T19:23:38.302757+0000","last_active":"2023-07-14T19:23:39.284243+0000","last_peered":"2023-07-14T19:23:39.284243+0000","last_clean":"2023-07-14T19:23:39.284243+0000","last_became_active":"2023-07-14T19:23:38.302409+0000","last_became_peered":"2023-07-14T19:23:38.302409+0000","last_unstale":"2023-07-14T19:23:39.284243+0000","last_undegraded":"2023-07-14T19:23:39.284243+0000","last_fullsized":"2023-07-14T19:23:39.284243+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:09:43.204311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515622+0000","last_change":"2023-07-14T19:23:38.296974+0000","last_active":"2023-07-14T19:23:39.515622+0000","last_peered":"2023-07-14T19:23:39.515622+0000","last_clean":"2023-07-14T19:23:39.515622+0000","last_became_active":"2023-07-14T19:23:38.296718+0000","last_became_peered":"2023-07-14T19:23:38.296718+0000","last_unstale":"2023-07-14T19:23:39.515622+0000","last_undegraded":"2023-07-14T19:23:39.515622+0000","last_fullsized":"2023-07-14T19:23:39.515622+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:19:13.680031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515564+0000","last_change":"2023-07-14T19:23:38.297575+0000","last_active":"2023-07-14T19:23:39.515564+0000","last_peered":"2023-07-14T19:23:39.515564+0000","last_clean":"2023-07-14T19:23:39.515564+0000","last_became_active":"2023-07-14T19:23:38.297292+0000","last_became_peered":"2023-07-14T19:23:38.297292+0000","last_unstale":"2023-07-14T19:23:39.515564+0000","last_undegraded":"2023-07-14T19:23:39.515564+0000","last_fullsized":"2023-07-14T19:23:39.515564+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:49.161960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515373+0000","last_change":"2023-07-14T19:23:38.295810+0000","last_active":"2023-07-14T19:23:39.515373+0000","last_peered":"2023-07-14T19:23:39.515373+0000","last_clean":"2023-07-14T19:23:39.515373+0000","last_became_active":"2023-07-14T19:23:38.295281+0000","last_became_peered":"2023-07-14T19:23:38.295281+0000","last_unstale":"2023-07-14T19:23:39.515373+0000","last_undegraded":"2023-07-14T19:23:39.515373+0000","last_fullsized":"2023-07-14T19:23:39.515373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:37.869421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514485+0000","last_change":"2023-07-14T19:23:38.299271+0000","last_active":"2023-07-14T19:23:39.514485+0000","last_peered":"2023-07-14T19:23:39.514485+0000","last_clean":"2023-07-14T19:23:39.514485+0000","last_became_active":"2023-07-14T19:23:38.298298+0000","last_became_peered":"2023-07-14T19:23:38.298298+0000","last_unstale":"2023-07-14T19:23:39.514485+0000","last_undegraded":"2023-07-14T19:23:39.514485+0000","last_fullsized":"2023-07-14T19:23:39.514485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:30.297961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514219+0000","last_change":"2023-07-14T19:23:38.296295+0000","last_active":"2023-07-14T19:23:39.514219+0000","last_peered":"2023-07-14T19:23:39.514219+0000","last_clean":"2023-07-14T19:23:39.514219+0000","last_became_active":"2023-07-14T19:23:38.296004+0000","last_became_peered":"2023-07-14T19:23:38.296004+0000","last_unstale":"2023-07-14T19:23:39.514219+0000","last_undegraded":"2023-07-14T19:23:39.514219+0000","last_fullsized":"2023-07-14T19:23:39.514219+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:34:50.891851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514438+0000","last_change":"2023-07-14T19:23:38.293292+0000","last_active":"2023-07-14T19:23:39.514438+0000","last_peered":"2023-07-14T19:23:39.514438+0000","last_clean":"2023-07-14T19:23:39.514438+0000","last_became_active":"2023-07-14T19:23:38.292980+0000","last_became_peered":"2023-07-14T19:23:38.292980+0000","last_unstale":"2023-07-14T19:23:39.514438+0000","last_undegraded":"2023-07-14T19:23:39.514438+0000","last_fullsized":"2023-07-14T19:23:39.514438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:22.060162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472781+0000","last_change":"2023-07-14T19:23:38.301950+0000","last_active":"2023-07-14T19:23:39.472781+0000","last_peered":"2023-07-14T19:23:39.472781+0000","last_clean":"2023-07-14T19:23:39.472781+0000","last_became_active":"2023-07-14T19:23:38.301416+0000","last_became_peered":"2023-07-14T19:23:38.301416+0000","last_unstale":"2023-07-14T19:23:39.472781+0000","last_undegraded":"2023-07-14T19:23:39.472781+0000","last_fullsized":"2023-07-14T19:23:39.472781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:03:11.369261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515639+0000","last_change":"2023-07-14T19:23:38.300108+0000","last_active":"2023-07-14T19:23:39.515639+0000","last_peered":"2023-07-14T19:23:39.515639+0000","last_clean":"2023-07-14T19:23:39.515639+0000","last_became_active":"2023-07-14T19:23:38.299773+0000","last_became_peered":"2023-07-14T19:23:38.299773+0000","last_unstale":"2023-07-14T19:23:39.515639+0000","last_undegraded":"2023-07-14T19:23:39.515639+0000","last_fullsized":"2023-07-14T19:23:39.515639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:59.607746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515238+0000","last_change":"2023-07-14T19:23:38.299124+0000","last_active":"2023-07-14T19:23:39.515238+0000","last_peered":"2023-07-14T19:23:39.515238+0000","last_clean":"2023-07-14T19:23:39.515238+0000","last_became_active":"2023-07-14T19:23:38.298744+0000","last_became_peered":"2023-07-14T19:23:38.298744+0000","last_unstale":"2023-07-14T19:23:39.515238+0000","last_undegraded":"2023-07-14T19:23:39.515238+0000","last_fullsized":"2023-07-14T19:23:39.515238+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:09.383666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"80'9","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.297045+0000","last_change":"2023-07-14T19:23:37.466299+0000","last_active":"2023-07-14T19:23:42.297045+0000","last_peered":"2023-07-14T19:23:42.297045+0000","last_clean":"2023-07-14T19:23:42.297045+0000","last_became_active":"2023-07-14T19:23:37.465989+0000","last_became_peered":"2023-07-14T19:23:37.465989+0000","last_unstale":"2023-07-14T19:23:42.297045+0000","last_undegraded":"2023-07-14T19:23:42.297045+0000","last_fullsized":"2023-07-14T19:23:42.297045+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:28.346196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283813+0000","last_change":"2023-07-14T19:23:38.301743+0000","last_active":"2023-07-14T19:23:39.283813+0000","last_peered":"2023-07-14T19:23:39.283813+0000","last_clean":"2023-07-14T19:23:39.283813+0000","last_became_active":"2023-07-14T19:23:38.301419+0000","last_became_peered":"2023-07-14T19:23:38.301419+0000","last_unstale":"2023-07-14T19:23:39.283813+0000","last_undegraded":"2023-07-14T19:23:39.283813+0000","last_fullsized":"2023-07-14T19:23:39.283813+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:37.577409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.263873+0000","last_change":"2023-07-14T19:23:37.459314+0000","last_active":"2023-07-14T19:23:41.263873+0000","last_peered":"2023-07-14T19:23:41.263873+0000","last_clean":"2023-07-14T19:23:41.263873+0000","last_became_active":"2023-07-14T19:23:37.458925+0000","last_became_peered":"2023-07-14T19:23:37.458925+0000","last_unstale":"2023-07-14T19:23:41.263873+0000","last_undegraded":"2023-07-14T19:23:41.263873+0000","last_fullsized":"2023-07-14T19:23:41.263873+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:59.317422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514675+0000","last_change":"2023-07-14T19:23:38.294432+0000","last_active":"2023-07-14T19:23:39.514675+0000","last_peered":"2023-07-14T19:23:39.514675+0000","last_clean":"2023-07-14T19:23:39.514675+0000","last_became_active":"2023-07-14T19:23:38.294118+0000","last_became_peered":"2023-07-14T19:23:38.294118+0000","last_unstale":"2023-07-14T19:23:39.514675+0000","last_undegraded":"2023-07-14T19:23:39.514675+0000","last_fullsized":"2023-07-14T19:23:39.514675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:48:03.277272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295257+0000","last_change":"2023-07-14T19:23:37.463231+0000","last_active":"2023-07-14T19:23:42.295257+0000","last_peered":"2023-07-14T19:23:42.295257+0000","last_clean":"2023-07-14T19:23:42.295257+0000","last_became_active":"2023-07-14T19:23:37.462571+0000","last_became_peered":"2023-07-14T19:23:37.462571+0000","last_unstale":"2023-07-14T19:23:42.295257+0000","last_undegraded":"2023-07-14T19:23:42.295257+0000","last_fullsized":"2023-07-14T19:23:42.295257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:07:42.060866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515814+0000","last_change":"2023-07-14T19:23:38.300276+0000","last_active":"2023-07-14T19:23:39.515814+0000","last_peered":"2023-07-14T19:23:39.515814+0000","last_clean":"2023-07-14T19:23:39.515814+0000","last_became_active":"2023-07-14T19:23:38.299984+0000","last_became_peered":"2023-07-14T19:23:38.299984+0000","last_unstale":"2023-07-14T19:23:39.515814+0000","last_undegraded":"2023-07-14T19:23:39.515814+0000","last_fullsized":"2023-07-14T19:23:39.515814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:19:10.316989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286669+0000","last_change":"2023-07-14T19:23:37.454592+0000","last_active":"2023-07-14T19:23:39.286669+0000","last_peered":"2023-07-14T19:23:39.286669+0000","last_clean":"2023-07-14T19:23:39.286669+0000","last_became_active":"2023-07-14T19:23:37.454266+0000","last_became_peered":"2023-07-14T19:23:37.454266+0000","last_unstale":"2023-07-14T19:23:39.286669+0000","last_undegraded":"2023-07-14T19:23:39.286669+0000","last_fullsized":"2023-07-14T19:23:39.286669+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:14.362148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.516093+0000","last_change":"2023-07-14T19:23:38.297945+0000","last_active":"2023-07-14T19:23:39.516093+0000","last_peered":"2023-07-14T19:23:39.516093+0000","last_clean":"2023-07-14T19:23:39.516093+0000","last_became_active":"2023-07-14T19:23:38.296733+0000","last_became_peered":"2023-07-14T19:23:38.296733+0000","last_unstale":"2023-07-14T19:23:39.516093+0000","last_undegraded":"2023-07-14T19:23:39.516093+0000","last_fullsized":"2023-07-14T19:23:39.516093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:38:24.778683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'6","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:21.354084+0000","last_change":"2023-07-14T19:23:37.460329+0000","last_active":"2023-07-14T19:24:21.354084+0000","last_peered":"2023-07-14T19:24:21.354084+0000","last_clean":"2023-07-14T19:24:21.354084+0000","last_became_active":"2023-07-14T19:23:37.460037+0000","last_became_peered":"2023-07-14T19:23:37.460037+0000","last_unstale":"2023-07-14T19:24:21.354084+0000","last_undegraded":"2023-07-14T19:24:21.354084+0000","last_fullsized":"2023-07-14T19:24:21.354084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:25:02.768120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286579+0000","last_change":"2023-07-14T19:23:38.293216+0000","last_active":"2023-07-14T19:23:39.286579+0000","last_peered":"2023-07-14T19:23:39.286579+0000","last_clean":"2023-07-14T19:23:39.286579+0000","last_became_active":"2023-07-14T19:23:38.292748+0000","last_became_peered":"2023-07-14T19:23:38.292748+0000","last_unstale":"2023-07-14T19:23:39.286579+0000","last_undegraded":"2023-07-14T19:23:39.286579+0000","last_fullsized":"2023-07-14T19:23:39.286579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:48.829793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295199+0000","last_change":"2023-07-14T19:23:37.462337+0000","last_active":"2023-07-14T19:23:42.295199+0000","last_peered":"2023-07-14T19:23:42.295199+0000","last_clean":"2023-07-14T19:23:42.295199+0000","last_became_active":"2023-07-14T19:23:37.462046+0000","last_became_peered":"2023-07-14T19:23:37.462046+0000","last_unstale":"2023-07-14T19:23:42.295199+0000","last_undegraded":"2023-07-14T19:23:42.295199+0000","last_fullsized":"2023-07-14T19:23:42.295199+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:07:29.186049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.516281+0000","last_change":"2023-07-14T19:23:38.297839+0000","last_active":"2023-07-14T19:23:39.516281+0000","last_peered":"2023-07-14T19:23:39.516281+0000","last_clean":"2023-07-14T19:23:39.516281+0000","last_became_active":"2023-07-14T19:23:38.296915+0000","last_became_peered":"2023-07-14T19:23:38.296915+0000","last_unstale":"2023-07-14T19:23:39.516281+0000","last_undegraded":"2023-07-14T19:23:39.516281+0000","last_fullsized":"2023-07-14T19:23:39.516281+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:44:44.107176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296502+0000","last_change":"2023-07-14T19:23:37.457637+0000","last_active":"2023-07-14T19:23:42.296502+0000","last_peered":"2023-07-14T19:23:42.296502+0000","last_clean":"2023-07-14T19:23:42.296502+0000","last_became_active":"2023-07-14T19:23:37.457305+0000","last_became_peered":"2023-07-14T19:23:37.457305+0000","last_unstale":"2023-07-14T19:23:42.296502+0000","last_undegraded":"2023-07-14T19:23:42.296502+0000","last_fullsized":"2023-07-14T19:23:42.296502+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:08:30.695027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283305+0000","last_change":"2023-07-14T19:23:38.295974+0000","last_active":"2023-07-14T19:23:39.283305+0000","last_peered":"2023-07-14T19:23:39.283305+0000","last_clean":"2023-07-14T19:23:39.283305+0000","last_became_active":"2023-07-14T19:23:38.295641+0000","last_became_peered":"2023-07-14T19:23:38.295641+0000","last_unstale":"2023-07-14T19:23:39.283305+0000","last_undegraded":"2023-07-14T19:23:39.283305+0000","last_fullsized":"2023-07-14T19:23:39.283305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:18:13.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515270+0000","last_change":"2023-07-14T19:23:37.457041+0000","last_active":"2023-07-14T19:23:39.515270+0000","last_peered":"2023-07-14T19:23:39.515270+0000","last_clean":"2023-07-14T19:23:39.515270+0000","last_became_active":"2023-07-14T19:23:37.456454+0000","last_became_peered":"2023-07-14T19:23:37.456454+0000","last_unstale":"2023-07-14T19:23:39.515270+0000","last_undegraded":"2023-07-14T19:23:39.515270+0000","last_fullsized":"2023-07-14T19:23:39.515270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:10:01.796045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515109+0000","last_change":"2023-07-14T19:23:38.293749+0000","last_active":"2023-07-14T19:23:39.515109+0000","last_peered":"2023-07-14T19:23:39.515109+0000","last_clean":"2023-07-14T19:23:39.515109+0000","last_became_active":"2023-07-14T19:23:38.293400+0000","last_became_peered":"2023-07-14T19:23:38.293400+0000","last_unstale":"2023-07-14T19:23:39.515109+0000","last_undegraded":"2023-07-14T19:23:39.515109+0000","last_fullsized":"2023-07-14T19:23:39.515109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:53.267258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'3","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:18.349018+0000","last_change":"2023-07-14T19:23:37.463783+0000","last_active":"2023-07-14T19:24:18.349018+0000","last_peered":"2023-07-14T19:24:18.349018+0000","last_clean":"2023-07-14T19:24:18.349018+0000","last_became_active":"2023-07-14T19:23:37.463421+0000","last_became_peered":"2023-07-14T19:23:37.463421+0000","last_unstale":"2023-07-14T19:24:18.349018+0000","last_undegraded":"2023-07-14T19:24:18.349018+0000","last_fullsized":"2023-07-14T19:24:18.349018+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:38.946539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515132+0000","last_change":"2023-07-14T19:23:38.300956+0000","last_active":"2023-07-14T19:23:39.515132+0000","last_peered":"2023-07-14T19:23:39.515132+0000","last_clean":"2023-07-14T19:23:39.515132+0000","last_became_active":"2023-07-14T19:23:38.300662+0000","last_became_peered":"2023-07-14T19:23:38.300662+0000","last_unstale":"2023-07-14T19:23:39.515132+0000","last_undegraded":"2023-07-14T19:23:39.515132+0000","last_fullsized":"2023-07-14T19:23:39.515132+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:59:21.546322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.476358+0000","last_change":"2023-07-14T19:23:37.458221+0000","last_active":"2023-07-14T19:23:41.476358+0000","last_peered":"2023-07-14T19:23:41.476358+0000","last_clean":"2023-07-14T19:23:41.476358+0000","last_became_active":"2023-07-14T19:23:37.457923+0000","last_became_peered":"2023-07-14T19:23:37.457923+0000","last_unstale":"2023-07-14T19:23:41.476358+0000","last_undegraded":"2023-07-14T19:23:41.476358+0000","last_fullsized":"2023-07-14T19:23:41.476358+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:51.733334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514538+0000","last_change":"2023-07-14T19:23:38.303977+0000","last_active":"2023-07-14T19:23:39.514538+0000","last_peered":"2023-07-14T19:23:39.514538+0000","last_clean":"2023-07-14T19:23:39.514538+0000","last_became_active":"2023-07-14T19:23:38.303333+0000","last_became_peered":"2023-07-14T19:23:38.303333+0000","last_unstale":"2023-07-14T19:23:39.514538+0000","last_undegraded":"2023-07-14T19:23:39.514538+0000","last_fullsized":"2023-07-14T19:23:39.514538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:28.958043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475704+0000","last_change":"2023-07-14T19:23:37.459181+0000","last_active":"2023-07-14T19:23:41.475704+0000","last_peered":"2023-07-14T19:23:41.475704+0000","last_clean":"2023-07-14T19:23:41.475704+0000","last_became_active":"2023-07-14T19:23:37.458890+0000","last_became_peered":"2023-07-14T19:23:37.458890+0000","last_unstale":"2023-07-14T19:23:41.475704+0000","last_undegraded":"2023-07-14T19:23:41.475704+0000","last_fullsized":"2023-07-14T19:23:41.475704+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:54.069559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515862+0000","last_change":"2023-07-14T19:23:38.295057+0000","last_active":"2023-07-14T19:23:39.515862+0000","last_peered":"2023-07-14T19:23:39.515862+0000","last_clean":"2023-07-14T19:23:39.515862+0000","last_became_active":"2023-07-14T19:23:38.294774+0000","last_became_peered":"2023-07-14T19:23:38.294774+0000","last_unstale":"2023-07-14T19:23:39.515862+0000","last_undegraded":"2023-07-14T19:23:39.515862+0000","last_fullsized":"2023-07-14T19:23:39.515862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:50:17.296593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'27","reported_seq":125,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:19.350464+0000","last_change":"2023-07-14T19:23:37.463196+0000","last_active":"2023-07-14T19:24:19.350464+0000","last_peered":"2023-07-14T19:24:19.350464+0000","last_clean":"2023-07-14T19:24:19.350464+0000","last_became_active":"2023-07-14T19:23:37.462867+0000","last_became_peered":"2023-07-14T19:23:37.462867+0000","last_unstale":"2023-07-14T19:24:19.350464+0000","last_undegraded":"2023-07-14T19:24:19.350464+0000","last_fullsized":"2023-07-14T19:24:19.350464+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:35.156443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58527,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":84,"num_read_kb":98,"num_write":28,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286612+0000","last_change":"2023-07-14T19:23:38.293249+0000","last_active":"2023-07-14T19:23:39.286612+0000","last_peered":"2023-07-14T19:23:39.286612+0000","last_clean":"2023-07-14T19:23:39.286612+0000","last_became_active":"2023-07-14T19:23:38.292938+0000","last_became_peered":"2023-07-14T19:23:38.292938+0000","last_unstale":"2023-07-14T19:23:39.286612+0000","last_undegraded":"2023-07-14T19:23:39.286612+0000","last_fullsized":"2023-07-14T19:23:39.286612+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:49:19.220723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"80'6","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296103+0000","last_change":"2023-07-14T19:23:37.460336+0000","last_active":"2023-07-14T19:23:42.296103+0000","last_peered":"2023-07-14T19:23:42.296103+0000","last_clean":"2023-07-14T19:23:42.296103+0000","last_became_active":"2023-07-14T19:23:37.460008+0000","last_became_peered":"2023-07-14T19:23:37.460008+0000","last_unstale":"2023-07-14T19:23:42.296103+0000","last_undegraded":"2023-07-14T19:23:42.296103+0000","last_fullsized":"2023-07-14T19:23:42.296103+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:31.610459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514712+0000","last_change":"2023-07-14T19:23:38.296819+0000","last_active":"2023-07-14T19:23:39.514712+0000","last_peered":"2023-07-14T19:23:39.514712+0000","last_clean":"2023-07-14T19:23:39.514712+0000","last_became_active":"2023-07-14T19:23:38.296225+0000","last_became_peered":"2023-07-14T19:23:38.296225+0000","last_unstale":"2023-07-14T19:23:39.514712+0000","last_undegraded":"2023-07-14T19:23:39.514712+0000","last_fullsized":"2023-07-14T19:23:39.514712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:43:30.925553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474332+0000","last_change":"2023-07-14T19:23:37.460018+0000","last_active":"2023-07-14T19:23:41.474332+0000","last_peered":"2023-07-14T19:23:41.474332+0000","last_clean":"2023-07-14T19:23:41.474332+0000","last_became_active":"2023-07-14T19:23:37.459713+0000","last_became_peered":"2023-07-14T19:23:37.459713+0000","last_unstale":"2023-07-14T19:23:41.474332+0000","last_undegraded":"2023-07-14T19:23:41.474332+0000","last_fullsized":"2023-07-14T19:23:41.474332+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:11.451517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515474+0000","last_change":"2023-07-14T19:23:38.301568+0000","last_active":"2023-07-14T19:23:39.515474+0000","last_peered":"2023-07-14T19:23:39.515474+0000","last_clean":"2023-07-14T19:23:39.515474+0000","last_became_active":"2023-07-14T19:23:38.301279+0000","last_became_peered":"2023-07-14T19:23:38.301279+0000","last_unstale":"2023-07-14T19:23:39.515474+0000","last_undegraded":"2023-07-14T19:23:39.515474+0000","last_fullsized":"2023-07-14T19:23:39.515474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:18.485184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'3","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:19.351129+0000","last_change":"2023-07-14T19:23:37.460782+0000","last_active":"2023-07-14T19:24:19.351129+0000","last_peered":"2023-07-14T19:24:19.351129+0000","last_clean":"2023-07-14T19:24:19.351129+0000","last_became_active":"2023-07-14T19:23:37.460457+0000","last_became_peered":"2023-07-14T19:23:37.460457+0000","last_unstale":"2023-07-14T19:24:19.351129+0000","last_undegraded":"2023-07-14T19:24:19.351129+0000","last_fullsized":"2023-07-14T19:24:19.351129+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:19.994091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514572+0000","last_change":"2023-07-14T19:23:38.299681+0000","last_active":"2023-07-14T19:23:39.514572+0000","last_peered":"2023-07-14T19:23:39.514572+0000","last_clean":"2023-07-14T19:23:39.514572+0000","last_became_active":"2023-07-14T19:23:38.299345+0000","last_became_peered":"2023-07-14T19:23:38.299345+0000","last_unstale":"2023-07-14T19:23:39.514572+0000","last_undegraded":"2023-07-14T19:23:39.514572+0000","last_fullsized":"2023-07-14T19:23:39.514572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:58:44.871954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"80'15","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296518+0000","last_change":"2023-07-14T19:23:37.458413+0000","last_active":"2023-07-14T19:23:42.296518+0000","last_peered":"2023-07-14T19:23:42.296518+0000","last_clean":"2023-07-14T19:23:42.296518+0000","last_became_active":"2023-07-14T19:23:37.458108+0000","last_became_peered":"2023-07-14T19:23:37.458108+0000","last_unstale":"2023-07-14T19:23:42.296518+0000","last_undegraded":"2023-07-14T19:23:42.296518+0000","last_fullsized":"2023-07-14T19:23:42.296518+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:36.538174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515638+0000","last_change":"2023-07-14T19:23:38.296350+0000","last_active":"2023-07-14T19:23:39.515638+0000","last_peered":"2023-07-14T19:23:39.515638+0000","last_clean":"2023-07-14T19:23:39.515638+0000","last_became_active":"2023-07-14T19:23:38.296036+0000","last_became_peered":"2023-07-14T19:23:38.296036+0000","last_unstale":"2023-07-14T19:23:39.515638+0000","last_undegraded":"2023-07-14T19:23:39.515638+0000","last_fullsized":"2023-07-14T19:23:39.515638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:58:58.339059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.286226+0000","last_change":"2023-07-14T19:23:37.461647+0000","last_active":"2023-07-14T19:23:42.286226+0000","last_peered":"2023-07-14T19:23:42.286226+0000","last_clean":"2023-07-14T19:23:42.286226+0000","last_became_active":"2023-07-14T19:23:37.461325+0000","last_became_peered":"2023-07-14T19:23:37.461325+0000","last_unstale":"2023-07-14T19:23:42.286226+0000","last_undegraded":"2023-07-14T19:23:42.286226+0000","last_fullsized":"2023-07-14T19:23:42.286226+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:46.235600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472796+0000","last_change":"2023-07-14T19:23:38.300940+0000","last_active":"2023-07-14T19:23:39.472796+0000","last_peered":"2023-07-14T19:23:39.472796+0000","last_clean":"2023-07-14T19:23:39.472796+0000","last_became_active":"2023-07-14T19:23:38.300578+0000","last_became_peered":"2023-07-14T19:23:38.300578+0000","last_unstale":"2023-07-14T19:23:39.472796+0000","last_undegraded":"2023-07-14T19:23:39.472796+0000","last_fullsized":"2023-07-14T19:23:39.472796+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:50.864591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515474+0000","last_change":"2023-07-14T19:23:37.464251+0000","last_active":"2023-07-14T19:23:39.515474+0000","last_peered":"2023-07-14T19:23:39.515474+0000","last_clean":"2023-07-14T19:23:39.515474+0000","last_became_active":"2023-07-14T19:23:37.463793+0000","last_became_peered":"2023-07-14T19:23:37.463793+0000","last_unstale":"2023-07-14T19:23:39.515474+0000","last_undegraded":"2023-07-14T19:23:39.515474+0000","last_fullsized":"2023-07-14T19:23:39.515474+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:28:41.697534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:40.299231+0000","last_change":"2023-07-14T19:23:38.300077+0000","last_active":"2023-07-14T19:23:40.299231+0000","last_peered":"2023-07-14T19:23:40.299231+0000","last_clean":"2023-07-14T19:23:40.299231+0000","last_became_active":"2023-07-14T19:23:38.298708+0000","last_became_peered":"2023-07-14T19:23:38.298708+0000","last_unstale":"2023-07-14T19:23:40.299231+0000","last_undegraded":"2023-07-14T19:23:40.299231+0000","last_fullsized":"2023-07-14T19:23:40.299231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:13:31.564130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515610+0000","last_change":"2023-07-14T19:23:37.458361+0000","last_active":"2023-07-14T19:23:39.515610+0000","last_peered":"2023-07-14T19:23:39.515610+0000","last_clean":"2023-07-14T19:23:39.515610+0000","last_became_active":"2023-07-14T19:23:37.458108+0000","last_became_peered":"2023-07-14T19:23:37.458108+0000","last_unstale":"2023-07-14T19:23:39.515610+0000","last_undegraded":"2023-07-14T19:23:39.515610+0000","last_fullsized":"2023-07-14T19:23:39.515610+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:41:17.858983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284311+0000","last_change":"2023-07-14T19:23:38.301723+0000","last_active":"2023-07-14T19:23:39.284311+0000","last_peered":"2023-07-14T19:23:39.284311+0000","last_clean":"2023-07-14T19:23:39.284311+0000","last_became_active":"2023-07-14T19:23:38.301424+0000","last_became_peered":"2023-07-14T19:23:38.301424+0000","last_unstale":"2023-07-14T19:23:39.284311+0000","last_undegraded":"2023-07-14T19:23:39.284311+0000","last_fullsized":"2023-07-14T19:23:39.284311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:49.753010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515589+0000","last_change":"2023-07-14T19:23:38.296954+0000","last_active":"2023-07-14T19:23:39.515589+0000","last_peered":"2023-07-14T19:23:39.515589+0000","last_clean":"2023-07-14T19:23:39.515589+0000","last_became_active":"2023-07-14T19:23:38.296143+0000","last_became_peered":"2023-07-14T19:23:38.296143+0000","last_unstale":"2023-07-14T19:23:39.515589+0000","last_undegraded":"2023-07-14T19:23:39.515589+0000","last_fullsized":"2023-07-14T19:23:39.515589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:58:27.181508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"80'4","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:17.347949+0000","last_change":"2023-07-14T19:23:37.460543+0000","last_active":"2023-07-14T19:24:17.347949+0000","last_peered":"2023-07-14T19:24:17.347949+0000","last_clean":"2023-07-14T19:24:17.347949+0000","last_became_active":"2023-07-14T19:23:37.460230+0000","last_became_peered":"2023-07-14T19:23:37.460230+0000","last_unstale":"2023-07-14T19:24:17.347949+0000","last_undegraded":"2023-07-14T19:24:17.347949+0000","last_fullsized":"2023-07-14T19:24:17.347949+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:17:14.842081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284020+0000","last_change":"2023-07-14T19:23:38.302414+0000","last_active":"2023-07-14T19:23:39.284020+0000","last_peered":"2023-07-14T19:23:39.284020+0000","last_clean":"2023-07-14T19:23:39.284020+0000","last_became_active":"2023-07-14T19:23:38.302089+0000","last_became_peered":"2023-07-14T19:23:38.302089+0000","last_unstale":"2023-07-14T19:23:39.284020+0000","last_undegraded":"2023-07-14T19:23:39.284020+0000","last_fullsized":"2023-07-14T19:23:39.284020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:09:15.082408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"80'3","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296431+0000","last_change":"2023-07-14T19:23:37.462245+0000","last_active":"2023-07-14T19:23:42.296431+0000","last_peered":"2023-07-14T19:23:42.296431+0000","last_clean":"2023-07-14T19:23:42.296431+0000","last_became_active":"2023-07-14T19:23:37.461911+0000","last_became_peered":"2023-07-14T19:23:37.461911+0000","last_unstale":"2023-07-14T19:23:42.296431+0000","last_undegraded":"2023-07-14T19:23:42.296431+0000","last_fullsized":"2023-07-14T19:23:42.296431+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:53.399246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":337,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515010+0000","last_change":"2023-07-14T19:22:39.254277+0000","last_active":"2023-07-14T19:23:39.515010+0000","last_peered":"2023-07-14T19:23:39.515010+0000","last_clean":"2023-07-14T19:23:39.515010+0000","last_became_active":"2023-07-14T19:22:39.253670+0000","last_became_peered":"2023-07-14T19:22:39.253670+0000","last_unstale":"2023-07-14T19:23:39.515010+0000","last_undegraded":"2023-07-14T19:23:39.515010+0000","last_fullsized":"2023-07-14T19:23:39.515010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:19:52.229492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:19:52.229492+0000","last_clean_scrub_stamp":"2023-07-14T19:19:52.229492+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:22.025952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514166+0000","last_change":"2023-07-14T19:23:38.296671+0000","last_active":"2023-07-14T19:23:39.514166+0000","last_peered":"2023-07-14T19:23:39.514166+0000","last_clean":"2023-07-14T19:23:39.514166+0000","last_became_active":"2023-07-14T19:23:38.296175+0000","last_became_peered":"2023-07-14T19:23:38.296175+0000","last_unstale":"2023-07-14T19:23:39.514166+0000","last_undegraded":"2023-07-14T19:23:39.514166+0000","last_fullsized":"2023-07-14T19:23:39.514166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:35:01.140553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295431+0000","last_change":"2023-07-14T19:23:37.459269+0000","last_active":"2023-07-14T19:23:42.295431+0000","last_peered":"2023-07-14T19:23:42.295431+0000","last_clean":"2023-07-14T19:23:42.295431+0000","last_became_active":"2023-07-14T19:23:37.458999+0000","last_became_peered":"2023-07-14T19:23:37.458999+0000","last_unstale":"2023-07-14T19:23:42.295431+0000","last_undegraded":"2023-07-14T19:23:42.295431+0000","last_fullsized":"2023-07-14T19:23:42.295431+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:45.895670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"80'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475684+0000","last_change":"2023-07-14T19:23:37.463425+0000","last_active":"2023-07-14T19:23:41.475684+0000","last_peered":"2023-07-14T19:23:41.475684+0000","last_clean":"2023-07-14T19:23:41.475684+0000","last_became_active":"2023-07-14T19:23:37.463005+0000","last_became_peered":"2023-07-14T19:23:37.463005+0000","last_unstale":"2023-07-14T19:23:41.475684+0000","last_undegraded":"2023-07-14T19:23:41.475684+0000","last_fullsized":"2023-07-14T19:23:41.475684+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:22:15.482538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515091+0000","last_change":"2023-07-14T19:23:38.295556+0000","last_active":"2023-07-14T19:23:39.515091+0000","last_peered":"2023-07-14T19:23:39.515091+0000","last_clean":"2023-07-14T19:23:39.515091+0000","last_became_active":"2023-07-14T19:23:38.295233+0000","last_became_peered":"2023-07-14T19:23:38.295233+0000","last_unstale":"2023-07-14T19:23:39.515091+0000","last_undegraded":"2023-07-14T19:23:39.515091+0000","last_fullsized":"2023-07-14T19:23:39.515091+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:29.024672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:46.782543+0000","last_change":"2023-07-14T19:23:37.464752+0000","last_active":"2023-07-14T19:23:46.782543+0000","last_peered":"2023-07-14T19:23:46.782543+0000","last_clean":"2023-07-14T19:23:46.782543+0000","last_became_active":"2023-07-14T19:23:37.464451+0000","last_became_peered":"2023-07-14T19:23:37.464451+0000","last_unstale":"2023-07-14T19:23:46.782543+0000","last_undegraded":"2023-07-14T19:23:46.782543+0000","last_fullsized":"2023-07-14T19:23:46.782543+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:30:13.403469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284070+0000","last_change":"2023-07-14T19:23:38.298475+0000","last_active":"2023-07-14T19:23:39.284070+0000","last_peered":"2023-07-14T19:23:39.284070+0000","last_clean":"2023-07-14T19:23:39.284070+0000","last_became_active":"2023-07-14T19:23:38.298141+0000","last_became_peered":"2023-07-14T19:23:38.298141+0000","last_unstale":"2023-07-14T19:23:39.284070+0000","last_undegraded":"2023-07-14T19:23:39.284070+0000","last_fullsized":"2023-07-14T19:23:39.284070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:02.834103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474300+0000","last_change":"2023-07-14T19:23:37.457182+0000","last_active":"2023-07-14T19:23:41.474300+0000","last_peered":"2023-07-14T19:23:41.474300+0000","last_clean":"2023-07-14T19:23:41.474300+0000","last_became_active":"2023-07-14T19:23:37.456825+0000","last_became_peered":"2023-07-14T19:23:37.456825+0000","last_unstale":"2023-07-14T19:23:41.474300+0000","last_undegraded":"2023-07-14T19:23:41.474300+0000","last_fullsized":"2023-07-14T19:23:41.474300+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:03.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514703+0000","last_change":"2023-07-14T19:23:38.303732+0000","last_active":"2023-07-14T19:23:39.514703+0000","last_peered":"2023-07-14T19:23:39.514703+0000","last_clean":"2023-07-14T19:23:39.514703+0000","last_became_active":"2023-07-14T19:23:38.303296+0000","last_became_peered":"2023-07-14T19:23:38.303296+0000","last_unstale":"2023-07-14T19:23:39.514703+0000","last_undegraded":"2023-07-14T19:23:39.514703+0000","last_fullsized":"2023-07-14T19:23:39.514703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:55:39.991626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475191+0000","last_change":"2023-07-14T19:23:37.461183+0000","last_active":"2023-07-14T19:23:41.475191+0000","last_peered":"2023-07-14T19:23:41.475191+0000","last_clean":"2023-07-14T19:23:41.475191+0000","last_became_active":"2023-07-14T19:23:37.460876+0000","last_became_peered":"2023-07-14T19:23:37.460876+0000","last_unstale":"2023-07-14T19:23:41.475191+0000","last_undegraded":"2023-07-14T19:23:41.475191+0000","last_fullsized":"2023-07-14T19:23:41.475191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:31.746901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286421+0000","last_change":"2023-07-14T19:23:38.292032+0000","last_active":"2023-07-14T19:23:39.286421+0000","last_peered":"2023-07-14T19:23:39.286421+0000","last_clean":"2023-07-14T19:23:39.286421+0000","last_became_active":"2023-07-14T19:23:38.291721+0000","last_became_peered":"2023-07-14T19:23:38.291721+0000","last_unstale":"2023-07-14T19:23:39.286421+0000","last_undegraded":"2023-07-14T19:23:39.286421+0000","last_fullsized":"2023-07-14T19:23:39.286421+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:58.334225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474819+0000","last_change":"2023-07-14T19:23:37.465597+0000","last_active":"2023-07-14T19:23:41.474819+0000","last_peered":"2023-07-14T19:23:41.474819+0000","last_clean":"2023-07-14T19:23:41.474819+0000","last_became_active":"2023-07-14T19:23:37.465233+0000","last_became_peered":"2023-07-14T19:23:37.465233+0000","last_unstale":"2023-07-14T19:23:41.474819+0000","last_undegraded":"2023-07-14T19:23:41.474819+0000","last_fullsized":"2023-07-14T19:23:41.474819+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:55.848783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472581+0000","last_change":"2023-07-14T19:23:38.295185+0000","last_active":"2023-07-14T19:23:39.472581+0000","last_peered":"2023-07-14T19:23:39.472581+0000","last_clean":"2023-07-14T19:23:39.472581+0000","last_became_active":"2023-07-14T19:23:38.294855+0000","last_became_peered":"2023-07-14T19:23:38.294855+0000","last_unstale":"2023-07-14T19:23:39.472581+0000","last_undegraded":"2023-07-14T19:23:39.472581+0000","last_fullsized":"2023-07-14T19:23:39.472581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:48.045146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:40.262742+0000","last_change":"2023-07-14T19:23:37.463517+0000","last_active":"2023-07-14T19:23:40.262742+0000","last_peered":"2023-07-14T19:23:40.262742+0000","last_clean":"2023-07-14T19:23:40.262742+0000","last_became_active":"2023-07-14T19:23:37.463158+0000","last_became_peered":"2023-07-14T19:23:37.463158+0000","last_unstale":"2023-07-14T19:23:40.262742+0000","last_undegraded":"2023-07-14T19:23:40.262742+0000","last_fullsized":"2023-07-14T19:23:40.262742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:01.142000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283721+0000","last_change":"2023-07-14T19:23:38.305375+0000","last_active":"2023-07-14T19:23:39.283721+0000","last_peered":"2023-07-14T19:23:39.283721+0000","last_clean":"2023-07-14T19:23:39.283721+0000","last_became_active":"2023-07-14T19:23:38.305084+0000","last_became_peered":"2023-07-14T19:23:38.305084+0000","last_unstale":"2023-07-14T19:23:39.283721+0000","last_undegraded":"2023-07-14T19:23:39.283721+0000","last_fullsized":"2023-07-14T19:23:39.283721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:08:51.652651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474944+0000","last_change":"2023-07-14T19:23:37.458672+0000","last_active":"2023-07-14T19:23:41.474944+0000","last_peered":"2023-07-14T19:23:41.474944+0000","last_clean":"2023-07-14T19:23:41.474944+0000","last_became_active":"2023-07-14T19:23:37.458359+0000","last_became_peered":"2023-07-14T19:23:37.458359+0000","last_unstale":"2023-07-14T19:23:41.474944+0000","last_undegraded":"2023-07-14T19:23:41.474944+0000","last_fullsized":"2023-07-14T19:23:41.474944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:00:25.922910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515725+0000","last_change":"2023-07-14T19:23:38.301741+0000","last_active":"2023-07-14T19:23:39.515725+0000","last_peered":"2023-07-14T19:23:39.515725+0000","last_clean":"2023-07-14T19:23:39.515725+0000","last_became_active":"2023-07-14T19:23:38.301430+0000","last_became_peered":"2023-07-14T19:23:38.301430+0000","last_unstale":"2023-07-14T19:23:39.515725+0000","last_undegraded":"2023-07-14T19:23:39.515725+0000","last_fullsized":"2023-07-14T19:23:39.515725+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:13.088226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296558+0000","last_change":"2023-07-14T19:23:37.461764+0000","last_active":"2023-07-14T19:23:42.296558+0000","last_peered":"2023-07-14T19:23:42.296558+0000","last_clean":"2023-07-14T19:23:42.296558+0000","last_became_active":"2023-07-14T19:23:37.460910+0000","last_became_peered":"2023-07-14T19:23:37.460910+0000","last_unstale":"2023-07-14T19:23:42.296558+0000","last_undegraded":"2023-07-14T19:23:42.296558+0000","last_fullsized":"2023-07-14T19:23:42.296558+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:39.478046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515703+0000","last_change":"2023-07-14T19:23:38.296214+0000","last_active":"2023-07-14T19:23:39.515703+0000","last_peered":"2023-07-14T19:23:39.515703+0000","last_clean":"2023-07-14T19:23:39.515703+0000","last_became_active":"2023-07-14T19:23:38.295148+0000","last_became_peered":"2023-07-14T19:23:38.295148+0000","last_unstale":"2023-07-14T19:23:39.515703+0000","last_undegraded":"2023-07-14T19:23:39.515703+0000","last_fullsized":"2023-07-14T19:23:39.515703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:30.117202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515939+0000","last_change":"2023-07-14T19:23:37.458245+0000","last_active":"2023-07-14T19:23:39.515939+0000","last_peered":"2023-07-14T19:23:39.515939+0000","last_clean":"2023-07-14T19:23:39.515939+0000","last_became_active":"2023-07-14T19:23:37.457784+0000","last_became_peered":"2023-07-14T19:23:37.457784+0000","last_unstale":"2023-07-14T19:23:39.515939+0000","last_undegraded":"2023-07-14T19:23:39.515939+0000","last_fullsized":"2023-07-14T19:23:39.515939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:47:00.389865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515370+0000","last_change":"2023-07-14T19:23:38.299823+0000","last_active":"2023-07-14T19:23:39.515370+0000","last_peered":"2023-07-14T19:23:39.515370+0000","last_clean":"2023-07-14T19:23:39.515370+0000","last_became_active":"2023-07-14T19:23:38.299496+0000","last_became_peered":"2023-07-14T19:23:38.299496+0000","last_unstale":"2023-07-14T19:23:39.515370+0000","last_undegraded":"2023-07-14T19:23:39.515370+0000","last_fullsized":"2023-07-14T19:23:39.515370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:23:40.353266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514282+0000","last_change":"2023-07-14T19:23:37.463439+0000","last_active":"2023-07-14T19:23:39.514282+0000","last_peered":"2023-07-14T19:23:39.514282+0000","last_clean":"2023-07-14T19:23:39.514282+0000","last_became_active":"2023-07-14T19:23:37.463100+0000","last_became_peered":"2023-07-14T19:23:37.463100+0000","last_unstale":"2023-07-14T19:23:39.514282+0000","last_undegraded":"2023-07-14T19:23:39.514282+0000","last_fullsized":"2023-07-14T19:23:39.514282+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:56.140535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515366+0000","last_change":"2023-07-14T19:23:38.296265+0000","last_active":"2023-07-14T19:23:39.515366+0000","last_peered":"2023-07-14T19:23:39.515366+0000","last_clean":"2023-07-14T19:23:39.515366+0000","last_became_active":"2023-07-14T19:23:38.295918+0000","last_became_peered":"2023-07-14T19:23:38.295918+0000","last_unstale":"2023-07-14T19:23:39.515366+0000","last_undegraded":"2023-07-14T19:23:39.515366+0000","last_fullsized":"2023-07-14T19:23:39.515366+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:38.769993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472522+0000","last_change":"2023-07-14T19:23:38.300200+0000","last_active":"2023-07-14T19:23:39.472522+0000","last_peered":"2023-07-14T19:23:39.472522+0000","last_clean":"2023-07-14T19:23:39.472522+0000","last_became_active":"2023-07-14T19:23:38.299979+0000","last_became_peered":"2023-07-14T19:23:38.299979+0000","last_unstale":"2023-07-14T19:23:39.472522+0000","last_undegraded":"2023-07-14T19:23:39.472522+0000","last_fullsized":"2023-07-14T19:23:39.472522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:00.789219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514842+0000","last_change":"2023-07-14T19:23:38.300696+0000","last_active":"2023-07-14T19:23:39.514842+0000","last_peered":"2023-07-14T19:23:39.514842+0000","last_clean":"2023-07-14T19:23:39.514842+0000","last_became_active":"2023-07-14T19:23:38.300377+0000","last_became_peered":"2023-07-14T19:23:38.300377+0000","last_unstale":"2023-07-14T19:23:39.514842+0000","last_undegraded":"2023-07-14T19:23:39.514842+0000","last_fullsized":"2023-07-14T19:23:39.514842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:39:05.440224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514814+0000","last_change":"2023-07-14T19:23:38.298616+0000","last_active":"2023-07-14T19:23:39.514814+0000","last_peered":"2023-07-14T19:23:39.514814+0000","last_clean":"2023-07-14T19:23:39.514814+0000","last_became_active":"2023-07-14T19:23:38.298313+0000","last_became_peered":"2023-07-14T19:23:38.298313+0000","last_unstale":"2023-07-14T19:23:39.514814+0000","last_undegraded":"2023-07-14T19:23:39.514814+0000","last_fullsized":"2023-07-14T19:23:39.514814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:46:49.950796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514982+0000","last_change":"2023-07-14T19:23:38.297185+0000","last_active":"2023-07-14T19:23:39.514982+0000","last_peered":"2023-07-14T19:23:39.514982+0000","last_clean":"2023-07-14T19:23:39.514982+0000","last_became_active":"2023-07-14T19:23:38.296883+0000","last_became_peered":"2023-07-14T19:23:38.296883+0000","last_unstale":"2023-07-14T19:23:39.514982+0000","last_undegraded":"2023-07-14T19:23:39.514982+0000","last_fullsized":"2023-07-14T19:23:39.514982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:08:27.715078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514907+0000","last_change":"2023-07-14T19:23:38.296636+0000","last_active":"2023-07-14T19:23:39.514907+0000","last_peered":"2023-07-14T19:23:39.514907+0000","last_clean":"2023-07-14T19:23:39.514907+0000","last_became_active":"2023-07-14T19:23:38.296289+0000","last_became_peered":"2023-07-14T19:23:38.296289+0000","last_unstale":"2023-07-14T19:23:39.514907+0000","last_undegraded":"2023-07-14T19:23:39.514907+0000","last_fullsized":"2023-07-14T19:23:39.514907+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:18:30.937689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472626+0000","last_change":"2023-07-14T19:23:38.300340+0000","last_active":"2023-07-14T19:23:39.472626+0000","last_peered":"2023-07-14T19:23:39.472626+0000","last_clean":"2023-07-14T19:23:39.472626+0000","last_became_active":"2023-07-14T19:23:38.299520+0000","last_became_peered":"2023-07-14T19:23:38.299520+0000","last_unstale":"2023-07-14T19:23:39.472626+0000","last_undegraded":"2023-07-14T19:23:39.472626+0000","last_fullsized":"2023-07-14T19:23:39.472626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:03:47.657130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515538+0000","last_change":"2023-07-14T19:23:38.294136+0000","last_active":"2023-07-14T19:23:39.515538+0000","last_peered":"2023-07-14T19:23:39.515538+0000","last_clean":"2023-07-14T19:23:39.515538+0000","last_became_active":"2023-07-14T19:23:38.293785+0000","last_became_peered":"2023-07-14T19:23:38.293785+0000","last_unstale":"2023-07-14T19:23:39.515538+0000","last_undegraded":"2023-07-14T19:23:39.515538+0000","last_fullsized":"2023-07-14T19:23:39.515538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:21.794486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284002+0000","last_change":"2023-07-14T19:23:38.305648+0000","last_active":"2023-07-14T19:23:39.284002+0000","last_peered":"2023-07-14T19:23:39.284002+0000","last_clean":"2023-07-14T19:23:39.284002+0000","last_became_active":"2023-07-14T19:23:38.305314+0000","last_became_peered":"2023-07-14T19:23:38.305314+0000","last_unstale":"2023-07-14T19:23:39.284002+0000","last_undegraded":"2023-07-14T19:23:39.284002+0000","last_fullsized":"2023-07-14T19:23:39.284002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:01:05.266523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515131+0000","last_change":"2023-07-14T19:23:38.298189+0000","last_active":"2023-07-14T19:23:39.515131+0000","last_peered":"2023-07-14T19:23:39.515131+0000","last_clean":"2023-07-14T19:23:39.515131+0000","last_became_active":"2023-07-14T19:23:38.297781+0000","last_became_peered":"2023-07-14T19:23:38.297781+0000","last_unstale":"2023-07-14T19:23:39.515131+0000","last_undegraded":"2023-07-14T19:23:39.515131+0000","last_fullsized":"2023-07-14T19:23:39.515131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:49:59.706932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283450+0000","last_change":"2023-07-14T19:23:38.293567+0000","last_active":"2023-07-14T19:23:39.283450+0000","last_peered":"2023-07-14T19:23:39.283450+0000","last_clean":"2023-07-14T19:23:39.283450+0000","last_became_active":"2023-07-14T19:23:38.293222+0000","last_became_peered":"2023-07-14T19:23:38.293222+0000","last_unstale":"2023-07-14T19:23:39.283450+0000","last_undegraded":"2023-07-14T19:23:39.283450+0000","last_fullsized":"2023-07-14T19:23:39.283450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:28.545521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515396+0000","last_change":"2023-07-14T19:23:38.294168+0000","last_active":"2023-07-14T19:23:39.515396+0000","last_peered":"2023-07-14T19:23:39.515396+0000","last_clean":"2023-07-14T19:23:39.515396+0000","last_became_active":"2023-07-14T19:23:38.293665+0000","last_became_peered":"2023-07-14T19:23:38.293665+0000","last_unstale":"2023-07-14T19:23:39.515396+0000","last_undegraded":"2023-07-14T19:23:39.515396+0000","last_fullsized":"2023-07-14T19:23:39.515396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:49.466175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515479+0000","last_change":"2023-07-14T19:23:38.296535+0000","last_active":"2023-07-14T19:23:39.515479+0000","last_peered":"2023-07-14T19:23:39.515479+0000","last_clean":"2023-07-14T19:23:39.515479+0000","last_became_active":"2023-07-14T19:23:38.296189+0000","last_became_peered":"2023-07-14T19:23:38.296189+0000","last_unstale":"2023-07-14T19:23:39.515479+0000","last_undegraded":"2023-07-14T19:23:39.515479+0000","last_fullsized":"2023-07-14T19:23:39.515479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:36:54.183312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472559+0000","last_change":"2023-07-14T19:23:38.296867+0000","last_active":"2023-07-14T19:23:39.472559+0000","last_peered":"2023-07-14T19:23:39.472559+0000","last_clean":"2023-07-14T19:23:39.472559+0000","last_became_active":"2023-07-14T19:23:38.296560+0000","last_became_peered":"2023-07-14T19:23:38.296560+0000","last_unstale":"2023-07-14T19:23:39.472559+0000","last_undegraded":"2023-07-14T19:23:39.472559+0000","last_fullsized":"2023-07-14T19:23:39.472559+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:48:16.893808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62431,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":256,"num_read_kb":191,"num_write":149,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":187293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547221,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":414159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.432}]},{"osd":1,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.55100000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.67700000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.74299999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.755},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.72499999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.80100000000000005}]}]},{"osd":10,"up_from":68,"seq":292057776151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1063794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.47299999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.66000000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.621},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.67500000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70999999999999996}]},{"osd":11,"last update":"Fri Jul 14 19:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.75},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.76100000000000001}]}]},{"osd":9,"up_from":63,"seq":270582939676,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":414395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.784,"5min":1.784,"15min":1.784},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.54400000000000004,"15min":0.56999999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.73799999999999999,"5min":2.6080000000000001,"15min":2.6080000000000001},"last":0.31900000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64000000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.42499999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80600000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.73,"5min":2.657,"15min":2.657},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.50900000000000001,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":0.434}]},{"osd":4,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67400000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.48799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.696,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":0.46700000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.76500000000000001,"15min":0.79900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.745,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.67500000000000004,"15min":0.68700000000000006},"min":{"1min":0.51600000000000001,"5min":0.499,"15min":0.499},"max":{"1min":0.83099999999999996,"5min":1.05,"15min":1.05},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.70199999999999996,"15min":0.71999999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.835,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.81399999999999995,"15min":0.84199999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.8560000000000001,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.59999999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.83599999999999997,"15min":0.86099999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.77400000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.73899999999999999,"15min":0.753},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.88200000000000001,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.504}]},{"osd":8,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.76600000000000001,"15min":0.77600000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.83399999999999996,"15min":0.86399999999999999},"min":{"1min":0.53800000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.78400000000000003,"5min":2.7309999999999999,"15min":2.7309999999999999},"last":0.66200000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":0.628}]}]},{"osd":8,"up_from":56,"seq":240518168608,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":413984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.504,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78800000000000003,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.46200000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.76100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.28399999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.503,"15min":0.505},"min":{"1min":0.29899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.48299999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59899999999999998,"15min":0.59999999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76400000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84099999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.39700000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83499999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.71499999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.39300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.88300000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.80100000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.746},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67200000000000004,"15min":0.67300000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.65400000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.73399999999999999,"15min":0.73099999999999998},"min":{"1min":0.56399999999999995,"5min":0.443,"15min":0.443},"max":{"1min":0.999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.871}]},{"osd":9,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68400000000000005,"15min":0.66900000000000004},"min":{"1min":0.59199999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.70999999999999996,"15min":0.69799999999999995},"min":{"1min":0.56200000000000006,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.81599999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.88500000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":7,"up_from":48,"seq":206158430244,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28456,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716696,"statfs":{"total":95995035648,"available":95965896704,"internally_reserved":0,"allocated":1474560,"data_stored":1006154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73099999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.82299999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.43099999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42499999999999999,"15min":0.42099999999999999},"min":{"1min":0.32900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78600000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.45800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.68899999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.58199999999999996}]},{"osd":4,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.502,"15min":0.49399999999999999},"min":{"1min":0.36299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.441}]},{"osd":5,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.621,"15min":0.624},"min":{"1min":0.47699999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66600000000000004,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.85799999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.51300000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66400000000000003},"min":{"1min":0.41199999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:23:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.67100000000000004,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.75800000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:24:18 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.58299999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.73399999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.83299999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]}]},{"osd":6,"up_from":43,"seq":184683593769,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":413790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.48799999999999999,"15min":0.432},"min":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":11.576000000000001,"5min":11.576000000000001,"15min":11.576000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.439,"15min":0.443},"min":{"1min":0.29499999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.41699999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.61499999999999999,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":11.202,"5min":11.202,"15min":11.202},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.999,"5min":0.66700000000000004,"15min":0.60199999999999998},"min":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":11.505000000000001,"5min":11.505000000000001,"15min":11.505000000000001},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.45200000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82199999999999995,"5min":1.071,"15min":1.071},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.71399999999999997,"15min":0.64600000000000002},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":11.390000000000001,"5min":11.390000000000001,"15min":11.390000000000001},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.58999999999999997,"15min":0.53600000000000003},"min":{"1min":0.34399999999999997,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":11.539999999999999,"5min":11.539999999999999,"15min":11.539999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.65400000000000003,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":11.680999999999999,"5min":11.680999999999999,"15min":11.680999999999999},"last":0.502},{"interface":"front","average":{"1min":1.022,"5min":0.73899999999999999,"15min":0.67300000000000004},"min":{"1min":0.47699999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":11.442,"5min":11.442,"15min":11.442},"last":0.71699999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.67100000000000004,"15min":0.66000000000000003},"min":{"1min":0.441,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86799999999999999,"5min":1.107,"15min":1.107},"last":0.622},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.69099999999999995,"15min":0.63700000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.747999999999999,"5min":11.747999999999999,"15min":11.747999999999999},"last":0.46600000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.70999999999999996}]},{"osd":8,"last update":"Fri Jul 14 19:23:56 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":0.79400000000000004,"15min":0.73799999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":11.778,"5min":11.778,"15min":11.778},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.21,"5min":0.875,"15min":0.81899999999999995},"min":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":11.929,"5min":11.929,"15min":11.929},"last":0.59199999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.84699999999999998,"15min":0.81399999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":11.888999999999999,"5min":11.888999999999999,"15min":11.888999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.77900000000000003,"15min":0.80400000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89100000000000001,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.65400000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.753}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":471831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53200000000000003,"15min":0.52200000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83999999999999997,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.59999999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.54300000000000004},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.56699999999999995,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.54700000000000004,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.878,"5min":1.028,"15min":1.028},"last":0.52400000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.40300000000000002},"min":{"1min":0.27600000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.433,"15min":0.42499999999999999},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.54300000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.84499999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.61699999999999999,"15min":0.60599999999999998},"min":{"1min":0.503,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.443},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.55500000000000005,"15min":0.54900000000000004},"min":{"1min":0.42599999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.61899999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.879},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.57099999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.68500000000000005,"15min":0.67700000000000005},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.75},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.71699999999999997,"15min":0.70499999999999996},"min":{"1min":0.52400000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0269999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73599999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.75800000000000001,"15min":0.746},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.042,"5min":1.079,"15min":1.079},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.68500000000000005,"15min":0.67500000000000004},"min":{"1min":0.59699999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.497}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.81000000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":415055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.52500000000000002,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.84599999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.51400000000000001,"15min":0.46800000000000003},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.51400000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65400000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54800000000000004,"15min":0.52500000000000002},"min":{"1min":0.371,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79400000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.71399999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.432,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.433},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.623},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59099999999999997,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.54700000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.45900000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78600000000000003,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60899999999999999,"15min":0.60399999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76300000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.748}]},{"osd":6,"last update":"Fri Jul 14 19:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67600000000000005,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.434,"15min":0.434},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.73099999999999998,"15min":0.72699999999999998},"min":{"1min":0.55900000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.58599999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.70899999999999996,"15min":0.72399999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.90600000000000003,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73599999999999999,"15min":0.746},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.997,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.73199999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.56299999999999994}]},{"osd":10,"last update":"Fri Jul 14 19:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":1.028},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.66000000000000003}]},{"osd":11,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.69999999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":473796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52700000000000002,"15min":0.51300000000000001},"min":{"1min":0.41399999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.505,"15min":0.49199999999999999},"min":{"1min":0.35999999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.72799999999999998,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58899999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53200000000000003,"15min":0.52100000000000002},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78600000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.622,"5min":0.55100000000000005,"15min":0.50900000000000001},"min":{"1min":0.45200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.63100000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57599999999999996,"15min":0.55400000000000005},"min":{"1min":0.36899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82899999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.623,"15min":0.57899999999999996},"min":{"1min":0.379,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.056,"5min":5.056,"15min":5.056},"last":0.51600000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.50700000000000001},"min":{"1min":0.438,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72599999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69299999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.72299999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.44600000000000001,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63600000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.76000000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63200000000000001,"15min":0.61599999999999999},"min":{"1min":0.42199999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.59399999999999997,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.69699999999999995}]},{"osd":7,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.434,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82799999999999996,"5min":3.6579999999999999,"15min":3.6579999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.79900000000000004,"15min":0.82899999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.79500000000000004,"5min":3.8769999999999998,"15min":3.8769999999999998},"last":0.45900000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:23:58 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69399999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68000000000000005,"15min":0.68600000000000005},"min":{"1min":0.40100000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.85199999999999998,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.81799999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.745,"15min":0.746},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.70399999999999996,"15min":0.69599999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.83399999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.88100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]}]},{"osd":3,"up_from":24,"seq":103079215157,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":414107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.34899999999999998},"min":{"1min":0.30399999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.39000000000000001,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.377}]},{"osd":1,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.54200000000000004,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.436,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.48999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.57899999999999996,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.46300000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.57699999999999996,"15min":0.55500000000000005},"min":{"1min":0.47299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.625,"15min":0.59999999999999998},"min":{"1min":0.41499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.67200000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.625,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.37,"15min":0.37},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.54000000000000004},"min":{"1min":0.36299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86599999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.503}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.443,"5min":0.436,"15min":0.436},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.73099999999999998,"15min":0.71699999999999997},"min":{"1min":0.54300000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.754}]},{"osd":8,"last update":"Fri Jul 14 19:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.54800000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.623}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.84799999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":414173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.34499999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.44,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73999999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.46200000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.36799999999999999,"15min":0.33800000000000002},"min":{"1min":0.29499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.78200000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.39700000000000002,"15min":0.377},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.81599999999999995,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.33400000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.51900000000000002},"min":{"1min":0.48799999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88900000000000001,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52600000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.40000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84799999999999998,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.5},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.54800000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59799999999999998,"15min":0.57799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.878,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.621,"15min":0.58099999999999996},"min":{"1min":0.55600000000000005,"5min":0.372,"15min":0.372},"max":{"1min":0.96099999999999997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.57199999999999995,"15min":0.56899999999999995},"min":{"1min":0.42299999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.94799999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.56799999999999995}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.53900000000000003,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.442,"5min":0.377,"15min":0.377},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.78500000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70999999999999996,"15min":0.70499999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.69099999999999995,"15min":0.68700000000000006},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.92500000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.64300000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":1.0169999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822701,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28464,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716688,"statfs":{"total":95995035648,"available":95965888512,"internally_reserved":0,"allocated":1482752,"data_stored":1005555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53400000000000003,"15min":0.51200000000000001},"min":{"1min":0.32900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.441},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53900000000000003,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53400000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.53700000000000003,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.40200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40100000000000002,"15min":0.39300000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56799999999999995,"15min":0.55700000000000005},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77500000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57499999999999996,"15min":0.56399999999999995},"min":{"1min":0.33500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77200000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.55800000000000005,"15min":0.54400000000000004},"min":{"1min":0.48599999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55600000000000005,"15min":0.53800000000000003},"min":{"1min":0.41199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63300000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.65000000000000002,"15min":0.625},"min":{"1min":0.53100000000000003,"5min":0.37,"15min":0.37},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.60099999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.51300000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64600000000000002,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.77900000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.59199999999999997,"15min":0.57899999999999996},"min":{"1min":0.38600000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.42399999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.72599999999999998,"15min":0.70699999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.71699999999999997,"15min":0.70999999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.65100000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:24:24.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:24:24.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:24 smithi053 ceph-mon[96565]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:24:24.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:24 smithi070 ceph-mon[98015]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:24:24.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:24 smithi104 ceph-mon[98435]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:24:24.844 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:25.600 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:25 smithi053 ceph-mon[96565]: pgmap v272: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:24:25.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:25 smithi070 ceph-mon[98015]: pgmap v272: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:24:25.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:25 smithi104 ceph-mon[98435]: pgmap v272: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:24:26.045 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.043+0000 7fd957850700 1 -- 172.21.15.53:0/1244634560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd950101150 msgr2=0x7fd95010bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:26.045 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.043+0000 7fd957850700 1 --2- 172.21.15.53:0/1244634560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd950101150 0x7fd95010bb40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd94800b0d0 tx=0x7fd94800b3e0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.043+0000 7fd957850700 1 -- 172.21.15.53:0/1244634560 shutdown_connections 2023-07-14T19:24:26.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.043+0000 7fd957850700 1 --2- 172.21.15.53:0/1244634560 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd950101150 0x7fd95010bb40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.043+0000 7fd957850700 1 --2- 172.21.15.53:0/1244634560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9500fef40 0x7fd9500ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.043+0000 7fd957850700 1 --2- 172.21.15.53:0/1244634560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9501007a0 0x7fd950100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.043+0000 7fd957850700 1 -- 172.21.15.53:0/1244634560 >> 172.21.15.53:0/1244634560 conn(0x7fd950075c90 msgr2=0x7fd950076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:26.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.043+0000 7fd957850700 1 -- 172.21.15.53:0/1244634560 shutdown_connections 2023-07-14T19:24:26.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.043+0000 7fd957850700 1 -- 172.21.15.53:0/1244634560 wait complete. 2023-07-14T19:24:26.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd957850700 1 Processor -- start 2023-07-14T19:24:26.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd957850700 1 -- start start 2023-07-14T19:24:26.048 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd957850700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9500fef40 0x7fd950196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:26.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd957850700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9501007a0 0x7fd950197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:26.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd957850700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd950101150 0x7fd9501a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:26.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd957850700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd950197c70 con 0x7fd9501007a0 2023-07-14T19:24:26.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd957850700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9501a23b0 con 0x7fd9500fef40 2023-07-14T19:24:26.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd957850700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9501a2570 con 0x7fd950101150 2023-07-14T19:24:26.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd954deb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9501007a0 0x7fd950197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:26.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd954deb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9501007a0 0x7fd950197330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50830/0 (socket says 172.21.15.53:50830) 2023-07-14T19:24:26.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.044+0000 7fd954deb700 1 -- 172.21.15.53:0/933374059 learned_addr learned my addr 172.21.15.53:0/933374059 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:26.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd955ded700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd950101150 0x7fd9501a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:26.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd9555ec700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9500fef40 0x7fd950196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:26.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd954deb700 1 -- 172.21.15.53:0/933374059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd950101150 msgr2=0x7fd9501a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:26.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd954deb700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd950101150 0x7fd9501a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd954deb700 1 -- 172.21.15.53:0/933374059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9500fef40 msgr2=0x7fd950196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:26.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd954deb700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9500fef40 0x7fd950196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd954deb700 1 -- 172.21.15.53:0/933374059 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd948009d00 con 0x7fd9501007a0 2023-07-14T19:24:26.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd954deb700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9501007a0 0x7fd950197330 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd94000c920 tx=0x7fd94000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:26.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd9467fc700 1 -- 172.21.15.53:0/933374059 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940007b00 con 0x7fd9501007a0 2023-07-14T19:24:26.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd9467fc700 1 -- 172.21.15.53:0/933374059 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd94000ce70 con 0x7fd9501007a0 2023-07-14T19:24:26.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9501a2850 con 0x7fd9501007a0 2023-07-14T19:24:26.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd9467fc700 1 -- 172.21.15.53:0/933374059 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400054e0 con 0x7fd9501007a0 2023-07-14T19:24:26.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.045+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9501a2da0 con 0x7fd9501007a0 2023-07-14T19:24:26.051 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.046+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95004e340 con 0x7fd9501007a0 2023-07-14T19:24:26.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.047+0000 7fd9467fc700 1 -- 172.21.15.53:0/933374059 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd940004750 con 0x7fd9501007a0 2023-07-14T19:24:26.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.048+0000 7fd9467fc700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd93c0a1260 0x7fd93c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:26.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.048+0000 7fd9555ec700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd93c0a1260 0x7fd93c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:26.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.048+0000 7fd9467fc700 1 -- 172.21.15.53:0/933374059 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd9400cc3f0 con 0x7fd9501007a0 2023-07-14T19:24:26.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.049+0000 7fd9555ec700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd93c0a1260 0x7fd93c0a3710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd94c006fd0 tx=0x7fd94c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:26.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.052+0000 7fd9467fc700 1 -- 172.21.15.53:0/933374059 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd940095b90 con 0x7fd9501007a0 2023-07-14T19:24:26.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.243+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9500fcc00 con 0x7fd93c0a1260 2023-07-14T19:24:26.256 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.255+0000 7fd9467fc700 1 -- 172.21.15.53:0/933374059 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+339846 (secure 0 0 0) 0x7fd9500fcc00 con 0x7fd93c0a1260 2023-07-14T19:24:26.260 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:26.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd93c0a1260 msgr2=0x7fd93c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:26.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd93c0a1260 0x7fd93c0a3710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd94c006fd0 tx=0x7fd94c008040 comp rx=0 tx=0).stop 2023-07-14T19:24:26.262 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9501007a0 msgr2=0x7fd950197330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:26.263 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9501007a0 0x7fd950197330 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd94000c920 tx=0x7fd94000cc30 comp rx=0 tx=0).stop 2023-07-14T19:24:26.263 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 shutdown_connections 2023-07-14T19:24:26.263 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd93c0a1260 0x7fd93c0a3710 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.263 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd950101150 0x7fd9501a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.264 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9500fef40 0x7fd950196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.264 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 --2- 172.21.15.53:0/933374059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9501007a0 0x7fd950197330 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.264 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.258+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 >> 172.21.15.53:0/933374059 conn(0x7fd950075c90 msgr2=0x7fd9500fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:26.265 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.259+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 shutdown_connections 2023-07-14T19:24:26.265 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:26.259+0000 7fd957850700 1 -- 172.21.15.53:0/933374059 wait complete. 2023-07-14T19:24:26.265 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2023-07-14T19:24:26.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:26 smithi053 ceph-mon[96565]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:24:26.705 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2023-07-14T19:24:25.326534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653045,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":412,"num_read_kb":325,"num_write":332,"num_write_kb":3042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270,"ondisk_log_size":270,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336428,"kb_used_data":12236,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605396,"statfs":{"total":1151940427776,"available":1151595925504,"internally_reserved":0,"allocated":12529664,"data_stored":6920793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":47,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003078"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472803+0000","last_change":"2023-07-14T19:23:38.301785+0000","last_active":"2023-07-14T19:23:39.472803+0000","last_peered":"2023-07-14T19:23:39.472803+0000","last_clean":"2023-07-14T19:23:39.472803+0000","last_became_active":"2023-07-14T19:23:38.301014+0000","last_became_peered":"2023-07-14T19:23:38.301014+0000","last_unstale":"2023-07-14T19:23:39.472803+0000","last_undegraded":"2023-07-14T19:23:39.472803+0000","last_fullsized":"2023-07-14T19:23:39.472803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:02.663606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514628+0000","last_change":"2023-07-14T19:23:38.299101+0000","last_active":"2023-07-14T19:23:39.514628+0000","last_peered":"2023-07-14T19:23:39.514628+0000","last_clean":"2023-07-14T19:23:39.514628+0000","last_became_active":"2023-07-14T19:23:38.298720+0000","last_became_peered":"2023-07-14T19:23:38.298720+0000","last_unstale":"2023-07-14T19:23:39.514628+0000","last_undegraded":"2023-07-14T19:23:39.514628+0000","last_fullsized":"2023-07-14T19:23:39.514628+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:05:31.873298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514167+0000","last_change":"2023-07-14T19:23:38.295335+0000","last_active":"2023-07-14T19:23:39.514167+0000","last_peered":"2023-07-14T19:23:39.514167+0000","last_clean":"2023-07-14T19:23:39.514167+0000","last_became_active":"2023-07-14T19:23:38.295031+0000","last_became_peered":"2023-07-14T19:23:38.295031+0000","last_unstale":"2023-07-14T19:23:39.514167+0000","last_undegraded":"2023-07-14T19:23:39.514167+0000","last_fullsized":"2023-07-14T19:23:39.514167+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:09.443353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515703+0000","last_change":"2023-07-14T19:23:38.295055+0000","last_active":"2023-07-14T19:23:39.515703+0000","last_peered":"2023-07-14T19:23:39.515703+0000","last_clean":"2023-07-14T19:23:39.515703+0000","last_became_active":"2023-07-14T19:23:38.294751+0000","last_became_peered":"2023-07-14T19:23:38.294751+0000","last_unstale":"2023-07-14T19:23:39.515703+0000","last_undegraded":"2023-07-14T19:23:39.515703+0000","last_fullsized":"2023-07-14T19:23:39.515703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:19.020190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286436+0000","last_change":"2023-07-14T19:23:38.294684+0000","last_active":"2023-07-14T19:23:39.286436+0000","last_peered":"2023-07-14T19:23:39.286436+0000","last_clean":"2023-07-14T19:23:39.286436+0000","last_became_active":"2023-07-14T19:23:38.294368+0000","last_became_peered":"2023-07-14T19:23:38.294368+0000","last_unstale":"2023-07-14T19:23:39.286436+0000","last_undegraded":"2023-07-14T19:23:39.286436+0000","last_fullsized":"2023-07-14T19:23:39.286436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:35.017660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283944+0000","last_change":"2023-07-14T19:23:38.301219+0000","last_active":"2023-07-14T19:23:39.283944+0000","last_peered":"2023-07-14T19:23:39.283944+0000","last_clean":"2023-07-14T19:23:39.283944+0000","last_became_active":"2023-07-14T19:23:38.300904+0000","last_became_peered":"2023-07-14T19:23:38.300904+0000","last_unstale":"2023-07-14T19:23:39.283944+0000","last_undegraded":"2023-07-14T19:23:39.283944+0000","last_fullsized":"2023-07-14T19:23:39.283944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:53:02.603728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515586+0000","last_change":"2023-07-14T19:23:38.292831+0000","last_active":"2023-07-14T19:23:39.515586+0000","last_peered":"2023-07-14T19:23:39.515586+0000","last_clean":"2023-07-14T19:23:39.515586+0000","last_became_active":"2023-07-14T19:23:38.292536+0000","last_became_peered":"2023-07-14T19:23:38.292536+0000","last_unstale":"2023-07-14T19:23:39.515586+0000","last_undegraded":"2023-07-14T19:23:39.515586+0000","last_fullsized":"2023-07-14T19:23:39.515586+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:49.316491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283393+0000","last_change":"2023-07-14T19:23:38.304940+0000","last_active":"2023-07-14T19:23:39.283393+0000","last_peered":"2023-07-14T19:23:39.283393+0000","last_clean":"2023-07-14T19:23:39.283393+0000","last_became_active":"2023-07-14T19:23:38.304609+0000","last_became_peered":"2023-07-14T19:23:38.304609+0000","last_unstale":"2023-07-14T19:23:39.283393+0000","last_undegraded":"2023-07-14T19:23:39.283393+0000","last_fullsized":"2023-07-14T19:23:39.283393+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:14.673980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515328+0000","last_change":"2023-07-14T19:23:38.300273+0000","last_active":"2023-07-14T19:23:39.515328+0000","last_peered":"2023-07-14T19:23:39.515328+0000","last_clean":"2023-07-14T19:23:39.515328+0000","last_became_active":"2023-07-14T19:23:38.299965+0000","last_became_peered":"2023-07-14T19:23:38.299965+0000","last_unstale":"2023-07-14T19:23:39.515328+0000","last_undegraded":"2023-07-14T19:23:39.515328+0000","last_fullsized":"2023-07-14T19:23:39.515328+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:26:21.737406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284243+0000","last_change":"2023-07-14T19:23:38.302757+0000","last_active":"2023-07-14T19:23:39.284243+0000","last_peered":"2023-07-14T19:23:39.284243+0000","last_clean":"2023-07-14T19:23:39.284243+0000","last_became_active":"2023-07-14T19:23:38.302409+0000","last_became_peered":"2023-07-14T19:23:38.302409+0000","last_unstale":"2023-07-14T19:23:39.284243+0000","last_undegraded":"2023-07-14T19:23:39.284243+0000","last_fullsized":"2023-07-14T19:23:39.284243+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:09:43.204311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515622+0000","last_change":"2023-07-14T19:23:38.296974+0000","last_active":"2023-07-14T19:23:39.515622+0000","last_peered":"2023-07-14T19:23:39.515622+0000","last_clean":"2023-07-14T19:23:39.515622+0000","last_became_active":"2023-07-14T19:23:38.296718+0000","last_became_peered":"2023-07-14T19:23:38.296718+0000","last_unstale":"2023-07-14T19:23:39.515622+0000","last_undegraded":"2023-07-14T19:23:39.515622+0000","last_fullsized":"2023-07-14T19:23:39.515622+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:19:13.680031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515564+0000","last_change":"2023-07-14T19:23:38.297575+0000","last_active":"2023-07-14T19:23:39.515564+0000","last_peered":"2023-07-14T19:23:39.515564+0000","last_clean":"2023-07-14T19:23:39.515564+0000","last_became_active":"2023-07-14T19:23:38.297292+0000","last_became_peered":"2023-07-14T19:23:38.297292+0000","last_unstale":"2023-07-14T19:23:39.515564+0000","last_undegraded":"2023-07-14T19:23:39.515564+0000","last_fullsized":"2023-07-14T19:23:39.515564+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:49.161960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515373+0000","last_change":"2023-07-14T19:23:38.295810+0000","last_active":"2023-07-14T19:23:39.515373+0000","last_peered":"2023-07-14T19:23:39.515373+0000","last_clean":"2023-07-14T19:23:39.515373+0000","last_became_active":"2023-07-14T19:23:38.295281+0000","last_became_peered":"2023-07-14T19:23:38.295281+0000","last_unstale":"2023-07-14T19:23:39.515373+0000","last_undegraded":"2023-07-14T19:23:39.515373+0000","last_fullsized":"2023-07-14T19:23:39.515373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:37.869421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514485+0000","last_change":"2023-07-14T19:23:38.299271+0000","last_active":"2023-07-14T19:23:39.514485+0000","last_peered":"2023-07-14T19:23:39.514485+0000","last_clean":"2023-07-14T19:23:39.514485+0000","last_became_active":"2023-07-14T19:23:38.298298+0000","last_became_peered":"2023-07-14T19:23:38.298298+0000","last_unstale":"2023-07-14T19:23:39.514485+0000","last_undegraded":"2023-07-14T19:23:39.514485+0000","last_fullsized":"2023-07-14T19:23:39.514485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:30.297961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514219+0000","last_change":"2023-07-14T19:23:38.296295+0000","last_active":"2023-07-14T19:23:39.514219+0000","last_peered":"2023-07-14T19:23:39.514219+0000","last_clean":"2023-07-14T19:23:39.514219+0000","last_became_active":"2023-07-14T19:23:38.296004+0000","last_became_peered":"2023-07-14T19:23:38.296004+0000","last_unstale":"2023-07-14T19:23:39.514219+0000","last_undegraded":"2023-07-14T19:23:39.514219+0000","last_fullsized":"2023-07-14T19:23:39.514219+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:34:50.891851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514438+0000","last_change":"2023-07-14T19:23:38.293292+0000","last_active":"2023-07-14T19:23:39.514438+0000","last_peered":"2023-07-14T19:23:39.514438+0000","last_clean":"2023-07-14T19:23:39.514438+0000","last_became_active":"2023-07-14T19:23:38.292980+0000","last_became_peered":"2023-07-14T19:23:38.292980+0000","last_unstale":"2023-07-14T19:23:39.514438+0000","last_undegraded":"2023-07-14T19:23:39.514438+0000","last_fullsized":"2023-07-14T19:23:39.514438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:22.060162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472781+0000","last_change":"2023-07-14T19:23:38.301950+0000","last_active":"2023-07-14T19:23:39.472781+0000","last_peered":"2023-07-14T19:23:39.472781+0000","last_clean":"2023-07-14T19:23:39.472781+0000","last_became_active":"2023-07-14T19:23:38.301416+0000","last_became_peered":"2023-07-14T19:23:38.301416+0000","last_unstale":"2023-07-14T19:23:39.472781+0000","last_undegraded":"2023-07-14T19:23:39.472781+0000","last_fullsized":"2023-07-14T19:23:39.472781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:03:11.369261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515639+0000","last_change":"2023-07-14T19:23:38.300108+0000","last_active":"2023-07-14T19:23:39.515639+0000","last_peered":"2023-07-14T19:23:39.515639+0000","last_clean":"2023-07-14T19:23:39.515639+0000","last_became_active":"2023-07-14T19:23:38.299773+0000","last_became_peered":"2023-07-14T19:23:38.299773+0000","last_unstale":"2023-07-14T19:23:39.515639+0000","last_undegraded":"2023-07-14T19:23:39.515639+0000","last_fullsized":"2023-07-14T19:23:39.515639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:59.607746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515238+0000","last_change":"2023-07-14T19:23:38.299124+0000","last_active":"2023-07-14T19:23:39.515238+0000","last_peered":"2023-07-14T19:23:39.515238+0000","last_clean":"2023-07-14T19:23:39.515238+0000","last_became_active":"2023-07-14T19:23:38.298744+0000","last_became_peered":"2023-07-14T19:23:38.298744+0000","last_unstale":"2023-07-14T19:23:39.515238+0000","last_undegraded":"2023-07-14T19:23:39.515238+0000","last_fullsized":"2023-07-14T19:23:39.515238+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:09.383666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"80'9","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.297045+0000","last_change":"2023-07-14T19:23:37.466299+0000","last_active":"2023-07-14T19:23:42.297045+0000","last_peered":"2023-07-14T19:23:42.297045+0000","last_clean":"2023-07-14T19:23:42.297045+0000","last_became_active":"2023-07-14T19:23:37.465989+0000","last_became_peered":"2023-07-14T19:23:37.465989+0000","last_unstale":"2023-07-14T19:23:42.297045+0000","last_undegraded":"2023-07-14T19:23:42.297045+0000","last_fullsized":"2023-07-14T19:23:42.297045+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:28.346196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283813+0000","last_change":"2023-07-14T19:23:38.301743+0000","last_active":"2023-07-14T19:23:39.283813+0000","last_peered":"2023-07-14T19:23:39.283813+0000","last_clean":"2023-07-14T19:23:39.283813+0000","last_became_active":"2023-07-14T19:23:38.301419+0000","last_became_peered":"2023-07-14T19:23:38.301419+0000","last_unstale":"2023-07-14T19:23:39.283813+0000","last_undegraded":"2023-07-14T19:23:39.283813+0000","last_fullsized":"2023-07-14T19:23:39.283813+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:37.577409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.263873+0000","last_change":"2023-07-14T19:23:37.459314+0000","last_active":"2023-07-14T19:23:41.263873+0000","last_peered":"2023-07-14T19:23:41.263873+0000","last_clean":"2023-07-14T19:23:41.263873+0000","last_became_active":"2023-07-14T19:23:37.458925+0000","last_became_peered":"2023-07-14T19:23:37.458925+0000","last_unstale":"2023-07-14T19:23:41.263873+0000","last_undegraded":"2023-07-14T19:23:41.263873+0000","last_fullsized":"2023-07-14T19:23:41.263873+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:59.317422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514675+0000","last_change":"2023-07-14T19:23:38.294432+0000","last_active":"2023-07-14T19:23:39.514675+0000","last_peered":"2023-07-14T19:23:39.514675+0000","last_clean":"2023-07-14T19:23:39.514675+0000","last_became_active":"2023-07-14T19:23:38.294118+0000","last_became_peered":"2023-07-14T19:23:38.294118+0000","last_unstale":"2023-07-14T19:23:39.514675+0000","last_undegraded":"2023-07-14T19:23:39.514675+0000","last_fullsized":"2023-07-14T19:23:39.514675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:48:03.277272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295257+0000","last_change":"2023-07-14T19:23:37.463231+0000","last_active":"2023-07-14T19:23:42.295257+0000","last_peered":"2023-07-14T19:23:42.295257+0000","last_clean":"2023-07-14T19:23:42.295257+0000","last_became_active":"2023-07-14T19:23:37.462571+0000","last_became_peered":"2023-07-14T19:23:37.462571+0000","last_unstale":"2023-07-14T19:23:42.295257+0000","last_undegraded":"2023-07-14T19:23:42.295257+0000","last_fullsized":"2023-07-14T19:23:42.295257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:07:42.060866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515814+0000","last_change":"2023-07-14T19:23:38.300276+0000","last_active":"2023-07-14T19:23:39.515814+0000","last_peered":"2023-07-14T19:23:39.515814+0000","last_clean":"2023-07-14T19:23:39.515814+0000","last_became_active":"2023-07-14T19:23:38.299984+0000","last_became_peered":"2023-07-14T19:23:38.299984+0000","last_unstale":"2023-07-14T19:23:39.515814+0000","last_undegraded":"2023-07-14T19:23:39.515814+0000","last_fullsized":"2023-07-14T19:23:39.515814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:19:10.316989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286669+0000","last_change":"2023-07-14T19:23:37.454592+0000","last_active":"2023-07-14T19:23:39.286669+0000","last_peered":"2023-07-14T19:23:39.286669+0000","last_clean":"2023-07-14T19:23:39.286669+0000","last_became_active":"2023-07-14T19:23:37.454266+0000","last_became_peered":"2023-07-14T19:23:37.454266+0000","last_unstale":"2023-07-14T19:23:39.286669+0000","last_undegraded":"2023-07-14T19:23:39.286669+0000","last_fullsized":"2023-07-14T19:23:39.286669+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:14.362148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.516093+0000","last_change":"2023-07-14T19:23:38.297945+0000","last_active":"2023-07-14T19:23:39.516093+0000","last_peered":"2023-07-14T19:23:39.516093+0000","last_clean":"2023-07-14T19:23:39.516093+0000","last_became_active":"2023-07-14T19:23:38.296733+0000","last_became_peered":"2023-07-14T19:23:38.296733+0000","last_unstale":"2023-07-14T19:23:39.516093+0000","last_undegraded":"2023-07-14T19:23:39.516093+0000","last_fullsized":"2023-07-14T19:23:39.516093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:38:24.778683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"80'6","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:21.354084+0000","last_change":"2023-07-14T19:23:37.460329+0000","last_active":"2023-07-14T19:24:21.354084+0000","last_peered":"2023-07-14T19:24:21.354084+0000","last_clean":"2023-07-14T19:24:21.354084+0000","last_became_active":"2023-07-14T19:23:37.460037+0000","last_became_peered":"2023-07-14T19:23:37.460037+0000","last_unstale":"2023-07-14T19:24:21.354084+0000","last_undegraded":"2023-07-14T19:24:21.354084+0000","last_fullsized":"2023-07-14T19:24:21.354084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:25:02.768120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286579+0000","last_change":"2023-07-14T19:23:38.293216+0000","last_active":"2023-07-14T19:23:39.286579+0000","last_peered":"2023-07-14T19:23:39.286579+0000","last_clean":"2023-07-14T19:23:39.286579+0000","last_became_active":"2023-07-14T19:23:38.292748+0000","last_became_peered":"2023-07-14T19:23:38.292748+0000","last_unstale":"2023-07-14T19:23:39.286579+0000","last_undegraded":"2023-07-14T19:23:39.286579+0000","last_fullsized":"2023-07-14T19:23:39.286579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:48.829793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295199+0000","last_change":"2023-07-14T19:23:37.462337+0000","last_active":"2023-07-14T19:23:42.295199+0000","last_peered":"2023-07-14T19:23:42.295199+0000","last_clean":"2023-07-14T19:23:42.295199+0000","last_became_active":"2023-07-14T19:23:37.462046+0000","last_became_peered":"2023-07-14T19:23:37.462046+0000","last_unstale":"2023-07-14T19:23:42.295199+0000","last_undegraded":"2023-07-14T19:23:42.295199+0000","last_fullsized":"2023-07-14T19:23:42.295199+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:07:29.186049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.516281+0000","last_change":"2023-07-14T19:23:38.297839+0000","last_active":"2023-07-14T19:23:39.516281+0000","last_peered":"2023-07-14T19:23:39.516281+0000","last_clean":"2023-07-14T19:23:39.516281+0000","last_became_active":"2023-07-14T19:23:38.296915+0000","last_became_peered":"2023-07-14T19:23:38.296915+0000","last_unstale":"2023-07-14T19:23:39.516281+0000","last_undegraded":"2023-07-14T19:23:39.516281+0000","last_fullsized":"2023-07-14T19:23:39.516281+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:44:44.107176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296502+0000","last_change":"2023-07-14T19:23:37.457637+0000","last_active":"2023-07-14T19:23:42.296502+0000","last_peered":"2023-07-14T19:23:42.296502+0000","last_clean":"2023-07-14T19:23:42.296502+0000","last_became_active":"2023-07-14T19:23:37.457305+0000","last_became_peered":"2023-07-14T19:23:37.457305+0000","last_unstale":"2023-07-14T19:23:42.296502+0000","last_undegraded":"2023-07-14T19:23:42.296502+0000","last_fullsized":"2023-07-14T19:23:42.296502+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:08:30.695027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283305+0000","last_change":"2023-07-14T19:23:38.295974+0000","last_active":"2023-07-14T19:23:39.283305+0000","last_peered":"2023-07-14T19:23:39.283305+0000","last_clean":"2023-07-14T19:23:39.283305+0000","last_became_active":"2023-07-14T19:23:38.295641+0000","last_became_peered":"2023-07-14T19:23:38.295641+0000","last_unstale":"2023-07-14T19:23:39.283305+0000","last_undegraded":"2023-07-14T19:23:39.283305+0000","last_fullsized":"2023-07-14T19:23:39.283305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:18:13.335760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515270+0000","last_change":"2023-07-14T19:23:37.457041+0000","last_active":"2023-07-14T19:23:39.515270+0000","last_peered":"2023-07-14T19:23:39.515270+0000","last_clean":"2023-07-14T19:23:39.515270+0000","last_became_active":"2023-07-14T19:23:37.456454+0000","last_became_peered":"2023-07-14T19:23:37.456454+0000","last_unstale":"2023-07-14T19:23:39.515270+0000","last_undegraded":"2023-07-14T19:23:39.515270+0000","last_fullsized":"2023-07-14T19:23:39.515270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:10:01.796045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515109+0000","last_change":"2023-07-14T19:23:38.293749+0000","last_active":"2023-07-14T19:23:39.515109+0000","last_peered":"2023-07-14T19:23:39.515109+0000","last_clean":"2023-07-14T19:23:39.515109+0000","last_became_active":"2023-07-14T19:23:38.293400+0000","last_became_peered":"2023-07-14T19:23:38.293400+0000","last_unstale":"2023-07-14T19:23:39.515109+0000","last_undegraded":"2023-07-14T19:23:39.515109+0000","last_fullsized":"2023-07-14T19:23:39.515109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:53.267258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'3","reported_seq":60,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:23.356665+0000","last_change":"2023-07-14T19:23:37.463783+0000","last_active":"2023-07-14T19:24:23.356665+0000","last_peered":"2023-07-14T19:24:23.356665+0000","last_clean":"2023-07-14T19:24:23.356665+0000","last_became_active":"2023-07-14T19:23:37.463421+0000","last_became_peered":"2023-07-14T19:23:37.463421+0000","last_unstale":"2023-07-14T19:24:23.356665+0000","last_undegraded":"2023-07-14T19:24:23.356665+0000","last_fullsized":"2023-07-14T19:24:23.356665+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:38.946539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515132+0000","last_change":"2023-07-14T19:23:38.300956+0000","last_active":"2023-07-14T19:23:39.515132+0000","last_peered":"2023-07-14T19:23:39.515132+0000","last_clean":"2023-07-14T19:23:39.515132+0000","last_became_active":"2023-07-14T19:23:38.300662+0000","last_became_peered":"2023-07-14T19:23:38.300662+0000","last_unstale":"2023-07-14T19:23:39.515132+0000","last_undegraded":"2023-07-14T19:23:39.515132+0000","last_fullsized":"2023-07-14T19:23:39.515132+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:59:21.546322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.476358+0000","last_change":"2023-07-14T19:23:37.458221+0000","last_active":"2023-07-14T19:23:41.476358+0000","last_peered":"2023-07-14T19:23:41.476358+0000","last_clean":"2023-07-14T19:23:41.476358+0000","last_became_active":"2023-07-14T19:23:37.457923+0000","last_became_peered":"2023-07-14T19:23:37.457923+0000","last_unstale":"2023-07-14T19:23:41.476358+0000","last_undegraded":"2023-07-14T19:23:41.476358+0000","last_fullsized":"2023-07-14T19:23:41.476358+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:51.733334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514538+0000","last_change":"2023-07-14T19:23:38.303977+0000","last_active":"2023-07-14T19:23:39.514538+0000","last_peered":"2023-07-14T19:23:39.514538+0000","last_clean":"2023-07-14T19:23:39.514538+0000","last_became_active":"2023-07-14T19:23:38.303333+0000","last_became_peered":"2023-07-14T19:23:38.303333+0000","last_unstale":"2023-07-14T19:23:39.514538+0000","last_undegraded":"2023-07-14T19:23:39.514538+0000","last_fullsized":"2023-07-14T19:23:39.514538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:28.958043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475704+0000","last_change":"2023-07-14T19:23:37.459181+0000","last_active":"2023-07-14T19:23:41.475704+0000","last_peered":"2023-07-14T19:23:41.475704+0000","last_clean":"2023-07-14T19:23:41.475704+0000","last_became_active":"2023-07-14T19:23:37.458890+0000","last_became_peered":"2023-07-14T19:23:37.458890+0000","last_unstale":"2023-07-14T19:23:41.475704+0000","last_undegraded":"2023-07-14T19:23:41.475704+0000","last_fullsized":"2023-07-14T19:23:41.475704+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:54.069559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515862+0000","last_change":"2023-07-14T19:23:38.295057+0000","last_active":"2023-07-14T19:23:39.515862+0000","last_peered":"2023-07-14T19:23:39.515862+0000","last_clean":"2023-07-14T19:23:39.515862+0000","last_became_active":"2023-07-14T19:23:38.294774+0000","last_became_peered":"2023-07-14T19:23:38.294774+0000","last_unstale":"2023-07-14T19:23:39.515862+0000","last_undegraded":"2023-07-14T19:23:39.515862+0000","last_fullsized":"2023-07-14T19:23:39.515862+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:50:17.296593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'27","reported_seq":135,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:24.358185+0000","last_change":"2023-07-14T19:23:37.463196+0000","last_active":"2023-07-14T19:24:24.358185+0000","last_peered":"2023-07-14T19:24:24.358185+0000","last_clean":"2023-07-14T19:24:24.358185+0000","last_became_active":"2023-07-14T19:23:37.462867+0000","last_became_peered":"2023-07-14T19:23:37.462867+0000","last_unstale":"2023-07-14T19:24:24.358185+0000","last_undegraded":"2023-07-14T19:24:24.358185+0000","last_fullsized":"2023-07-14T19:24:24.358185+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:35.156443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58527,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":94,"num_read_kb":103,"num_write":28,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286612+0000","last_change":"2023-07-14T19:23:38.293249+0000","last_active":"2023-07-14T19:23:39.286612+0000","last_peered":"2023-07-14T19:23:39.286612+0000","last_clean":"2023-07-14T19:23:39.286612+0000","last_became_active":"2023-07-14T19:23:38.292938+0000","last_became_peered":"2023-07-14T19:23:38.292938+0000","last_unstale":"2023-07-14T19:23:39.286612+0000","last_undegraded":"2023-07-14T19:23:39.286612+0000","last_fullsized":"2023-07-14T19:23:39.286612+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:49:19.220723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"80'6","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296103+0000","last_change":"2023-07-14T19:23:37.460336+0000","last_active":"2023-07-14T19:23:42.296103+0000","last_peered":"2023-07-14T19:23:42.296103+0000","last_clean":"2023-07-14T19:23:42.296103+0000","last_became_active":"2023-07-14T19:23:37.460008+0000","last_became_peered":"2023-07-14T19:23:37.460008+0000","last_unstale":"2023-07-14T19:23:42.296103+0000","last_undegraded":"2023-07-14T19:23:42.296103+0000","last_fullsized":"2023-07-14T19:23:42.296103+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:31.610459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514712+0000","last_change":"2023-07-14T19:23:38.296819+0000","last_active":"2023-07-14T19:23:39.514712+0000","last_peered":"2023-07-14T19:23:39.514712+0000","last_clean":"2023-07-14T19:23:39.514712+0000","last_became_active":"2023-07-14T19:23:38.296225+0000","last_became_peered":"2023-07-14T19:23:38.296225+0000","last_unstale":"2023-07-14T19:23:39.514712+0000","last_undegraded":"2023-07-14T19:23:39.514712+0000","last_fullsized":"2023-07-14T19:23:39.514712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:43:30.925553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474332+0000","last_change":"2023-07-14T19:23:37.460018+0000","last_active":"2023-07-14T19:23:41.474332+0000","last_peered":"2023-07-14T19:23:41.474332+0000","last_clean":"2023-07-14T19:23:41.474332+0000","last_became_active":"2023-07-14T19:23:37.459713+0000","last_became_peered":"2023-07-14T19:23:37.459713+0000","last_unstale":"2023-07-14T19:23:41.474332+0000","last_undegraded":"2023-07-14T19:23:41.474332+0000","last_fullsized":"2023-07-14T19:23:41.474332+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:11.451517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515474+0000","last_change":"2023-07-14T19:23:38.301568+0000","last_active":"2023-07-14T19:23:39.515474+0000","last_peered":"2023-07-14T19:23:39.515474+0000","last_clean":"2023-07-14T19:23:39.515474+0000","last_became_active":"2023-07-14T19:23:38.301279+0000","last_became_peered":"2023-07-14T19:23:38.301279+0000","last_unstale":"2023-07-14T19:23:39.515474+0000","last_undegraded":"2023-07-14T19:23:39.515474+0000","last_fullsized":"2023-07-14T19:23:39.515474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:18.485184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'3","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:19.351129+0000","last_change":"2023-07-14T19:23:37.460782+0000","last_active":"2023-07-14T19:24:19.351129+0000","last_peered":"2023-07-14T19:24:19.351129+0000","last_clean":"2023-07-14T19:24:19.351129+0000","last_became_active":"2023-07-14T19:23:37.460457+0000","last_became_peered":"2023-07-14T19:23:37.460457+0000","last_unstale":"2023-07-14T19:24:19.351129+0000","last_undegraded":"2023-07-14T19:24:19.351129+0000","last_fullsized":"2023-07-14T19:24:19.351129+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:19.994091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514572+0000","last_change":"2023-07-14T19:23:38.299681+0000","last_active":"2023-07-14T19:23:39.514572+0000","last_peered":"2023-07-14T19:23:39.514572+0000","last_clean":"2023-07-14T19:23:39.514572+0000","last_became_active":"2023-07-14T19:23:38.299345+0000","last_became_peered":"2023-07-14T19:23:38.299345+0000","last_unstale":"2023-07-14T19:23:39.514572+0000","last_undegraded":"2023-07-14T19:23:39.514572+0000","last_fullsized":"2023-07-14T19:23:39.514572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:58:44.871954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"80'15","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296518+0000","last_change":"2023-07-14T19:23:37.458413+0000","last_active":"2023-07-14T19:23:42.296518+0000","last_peered":"2023-07-14T19:23:42.296518+0000","last_clean":"2023-07-14T19:23:42.296518+0000","last_became_active":"2023-07-14T19:23:37.458108+0000","last_became_peered":"2023-07-14T19:23:37.458108+0000","last_unstale":"2023-07-14T19:23:42.296518+0000","last_undegraded":"2023-07-14T19:23:42.296518+0000","last_fullsized":"2023-07-14T19:23:42.296518+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:36.538174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515638+0000","last_change":"2023-07-14T19:23:38.296350+0000","last_active":"2023-07-14T19:23:39.515638+0000","last_peered":"2023-07-14T19:23:39.515638+0000","last_clean":"2023-07-14T19:23:39.515638+0000","last_became_active":"2023-07-14T19:23:38.296036+0000","last_became_peered":"2023-07-14T19:23:38.296036+0000","last_unstale":"2023-07-14T19:23:39.515638+0000","last_undegraded":"2023-07-14T19:23:39.515638+0000","last_fullsized":"2023-07-14T19:23:39.515638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:58:58.339059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'2","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.286226+0000","last_change":"2023-07-14T19:23:37.461647+0000","last_active":"2023-07-14T19:23:42.286226+0000","last_peered":"2023-07-14T19:23:42.286226+0000","last_clean":"2023-07-14T19:23:42.286226+0000","last_became_active":"2023-07-14T19:23:37.461325+0000","last_became_peered":"2023-07-14T19:23:37.461325+0000","last_unstale":"2023-07-14T19:23:42.286226+0000","last_undegraded":"2023-07-14T19:23:42.286226+0000","last_fullsized":"2023-07-14T19:23:42.286226+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:46.235600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472796+0000","last_change":"2023-07-14T19:23:38.300940+0000","last_active":"2023-07-14T19:23:39.472796+0000","last_peered":"2023-07-14T19:23:39.472796+0000","last_clean":"2023-07-14T19:23:39.472796+0000","last_became_active":"2023-07-14T19:23:38.300578+0000","last_became_peered":"2023-07-14T19:23:38.300578+0000","last_unstale":"2023-07-14T19:23:39.472796+0000","last_undegraded":"2023-07-14T19:23:39.472796+0000","last_fullsized":"2023-07-14T19:23:39.472796+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:50.864591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515474+0000","last_change":"2023-07-14T19:23:37.464251+0000","last_active":"2023-07-14T19:23:39.515474+0000","last_peered":"2023-07-14T19:23:39.515474+0000","last_clean":"2023-07-14T19:23:39.515474+0000","last_became_active":"2023-07-14T19:23:37.463793+0000","last_became_peered":"2023-07-14T19:23:37.463793+0000","last_unstale":"2023-07-14T19:23:39.515474+0000","last_undegraded":"2023-07-14T19:23:39.515474+0000","last_fullsized":"2023-07-14T19:23:39.515474+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:28:41.697534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:40.299231+0000","last_change":"2023-07-14T19:23:38.300077+0000","last_active":"2023-07-14T19:23:40.299231+0000","last_peered":"2023-07-14T19:23:40.299231+0000","last_clean":"2023-07-14T19:23:40.299231+0000","last_became_active":"2023-07-14T19:23:38.298708+0000","last_became_peered":"2023-07-14T19:23:38.298708+0000","last_unstale":"2023-07-14T19:23:40.299231+0000","last_undegraded":"2023-07-14T19:23:40.299231+0000","last_fullsized":"2023-07-14T19:23:40.299231+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:13:31.564130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515610+0000","last_change":"2023-07-14T19:23:37.458361+0000","last_active":"2023-07-14T19:23:39.515610+0000","last_peered":"2023-07-14T19:23:39.515610+0000","last_clean":"2023-07-14T19:23:39.515610+0000","last_became_active":"2023-07-14T19:23:37.458108+0000","last_became_peered":"2023-07-14T19:23:37.458108+0000","last_unstale":"2023-07-14T19:23:39.515610+0000","last_undegraded":"2023-07-14T19:23:39.515610+0000","last_fullsized":"2023-07-14T19:23:39.515610+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:41:17.858983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284311+0000","last_change":"2023-07-14T19:23:38.301723+0000","last_active":"2023-07-14T19:23:39.284311+0000","last_peered":"2023-07-14T19:23:39.284311+0000","last_clean":"2023-07-14T19:23:39.284311+0000","last_became_active":"2023-07-14T19:23:38.301424+0000","last_became_peered":"2023-07-14T19:23:38.301424+0000","last_unstale":"2023-07-14T19:23:39.284311+0000","last_undegraded":"2023-07-14T19:23:39.284311+0000","last_fullsized":"2023-07-14T19:23:39.284311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:49.753010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515589+0000","last_change":"2023-07-14T19:23:38.296954+0000","last_active":"2023-07-14T19:23:39.515589+0000","last_peered":"2023-07-14T19:23:39.515589+0000","last_clean":"2023-07-14T19:23:39.515589+0000","last_became_active":"2023-07-14T19:23:38.296143+0000","last_became_peered":"2023-07-14T19:23:38.296143+0000","last_unstale":"2023-07-14T19:23:39.515589+0000","last_undegraded":"2023-07-14T19:23:39.515589+0000","last_fullsized":"2023-07-14T19:23:39.515589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:58:27.181508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"80'4","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:24:22.354889+0000","last_change":"2023-07-14T19:23:37.460543+0000","last_active":"2023-07-14T19:24:22.354889+0000","last_peered":"2023-07-14T19:24:22.354889+0000","last_clean":"2023-07-14T19:24:22.354889+0000","last_became_active":"2023-07-14T19:23:37.460230+0000","last_became_peered":"2023-07-14T19:23:37.460230+0000","last_unstale":"2023-07-14T19:24:22.354889+0000","last_undegraded":"2023-07-14T19:24:22.354889+0000","last_fullsized":"2023-07-14T19:24:22.354889+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:17:14.842081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284020+0000","last_change":"2023-07-14T19:23:38.302414+0000","last_active":"2023-07-14T19:23:39.284020+0000","last_peered":"2023-07-14T19:23:39.284020+0000","last_clean":"2023-07-14T19:23:39.284020+0000","last_became_active":"2023-07-14T19:23:38.302089+0000","last_became_peered":"2023-07-14T19:23:38.302089+0000","last_unstale":"2023-07-14T19:23:39.284020+0000","last_undegraded":"2023-07-14T19:23:39.284020+0000","last_fullsized":"2023-07-14T19:23:39.284020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:09:15.082408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"80'3","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296431+0000","last_change":"2023-07-14T19:23:37.462245+0000","last_active":"2023-07-14T19:23:42.296431+0000","last_peered":"2023-07-14T19:23:42.296431+0000","last_clean":"2023-07-14T19:23:42.296431+0000","last_became_active":"2023-07-14T19:23:37.461911+0000","last_became_peered":"2023-07-14T19:23:37.461911+0000","last_unstale":"2023-07-14T19:23:42.296431+0000","last_undegraded":"2023-07-14T19:23:42.296431+0000","last_fullsized":"2023-07-14T19:23:42.296431+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:53.399246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":337,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515010+0000","last_change":"2023-07-14T19:22:39.254277+0000","last_active":"2023-07-14T19:23:39.515010+0000","last_peered":"2023-07-14T19:23:39.515010+0000","last_clean":"2023-07-14T19:23:39.515010+0000","last_became_active":"2023-07-14T19:22:39.253670+0000","last_became_peered":"2023-07-14T19:22:39.253670+0000","last_unstale":"2023-07-14T19:23:39.515010+0000","last_undegraded":"2023-07-14T19:23:39.515010+0000","last_fullsized":"2023-07-14T19:23:39.515010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:19:52.229492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:19:52.229492+0000","last_clean_scrub_stamp":"2023-07-14T19:19:52.229492+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:22.025952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514166+0000","last_change":"2023-07-14T19:23:38.296671+0000","last_active":"2023-07-14T19:23:39.514166+0000","last_peered":"2023-07-14T19:23:39.514166+0000","last_clean":"2023-07-14T19:23:39.514166+0000","last_became_active":"2023-07-14T19:23:38.296175+0000","last_became_peered":"2023-07-14T19:23:38.296175+0000","last_unstale":"2023-07-14T19:23:39.514166+0000","last_undegraded":"2023-07-14T19:23:39.514166+0000","last_fullsized":"2023-07-14T19:23:39.514166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:35:01.140553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.295431+0000","last_change":"2023-07-14T19:23:37.459269+0000","last_active":"2023-07-14T19:23:42.295431+0000","last_peered":"2023-07-14T19:23:42.295431+0000","last_clean":"2023-07-14T19:23:42.295431+0000","last_became_active":"2023-07-14T19:23:37.458999+0000","last_became_peered":"2023-07-14T19:23:37.458999+0000","last_unstale":"2023-07-14T19:23:42.295431+0000","last_undegraded":"2023-07-14T19:23:42.295431+0000","last_fullsized":"2023-07-14T19:23:42.295431+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:45.895670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"80'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475684+0000","last_change":"2023-07-14T19:23:37.463425+0000","last_active":"2023-07-14T19:23:41.475684+0000","last_peered":"2023-07-14T19:23:41.475684+0000","last_clean":"2023-07-14T19:23:41.475684+0000","last_became_active":"2023-07-14T19:23:37.463005+0000","last_became_peered":"2023-07-14T19:23:37.463005+0000","last_unstale":"2023-07-14T19:23:41.475684+0000","last_undegraded":"2023-07-14T19:23:41.475684+0000","last_fullsized":"2023-07-14T19:23:41.475684+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:22:15.482538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515091+0000","last_change":"2023-07-14T19:23:38.295556+0000","last_active":"2023-07-14T19:23:39.515091+0000","last_peered":"2023-07-14T19:23:39.515091+0000","last_clean":"2023-07-14T19:23:39.515091+0000","last_became_active":"2023-07-14T19:23:38.295233+0000","last_became_peered":"2023-07-14T19:23:38.295233+0000","last_unstale":"2023-07-14T19:23:39.515091+0000","last_undegraded":"2023-07-14T19:23:39.515091+0000","last_fullsized":"2023-07-14T19:23:39.515091+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:29.024672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'5","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:46.782543+0000","last_change":"2023-07-14T19:23:37.464752+0000","last_active":"2023-07-14T19:23:46.782543+0000","last_peered":"2023-07-14T19:23:46.782543+0000","last_clean":"2023-07-14T19:23:46.782543+0000","last_became_active":"2023-07-14T19:23:37.464451+0000","last_became_peered":"2023-07-14T19:23:37.464451+0000","last_unstale":"2023-07-14T19:23:46.782543+0000","last_undegraded":"2023-07-14T19:23:46.782543+0000","last_fullsized":"2023-07-14T19:23:46.782543+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:30:13.403469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284070+0000","last_change":"2023-07-14T19:23:38.298475+0000","last_active":"2023-07-14T19:23:39.284070+0000","last_peered":"2023-07-14T19:23:39.284070+0000","last_clean":"2023-07-14T19:23:39.284070+0000","last_became_active":"2023-07-14T19:23:38.298141+0000","last_became_peered":"2023-07-14T19:23:38.298141+0000","last_unstale":"2023-07-14T19:23:39.284070+0000","last_undegraded":"2023-07-14T19:23:39.284070+0000","last_fullsized":"2023-07-14T19:23:39.284070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:02.834103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"80'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474300+0000","last_change":"2023-07-14T19:23:37.457182+0000","last_active":"2023-07-14T19:23:41.474300+0000","last_peered":"2023-07-14T19:23:41.474300+0000","last_clean":"2023-07-14T19:23:41.474300+0000","last_became_active":"2023-07-14T19:23:37.456825+0000","last_became_peered":"2023-07-14T19:23:37.456825+0000","last_unstale":"2023-07-14T19:23:41.474300+0000","last_undegraded":"2023-07-14T19:23:41.474300+0000","last_fullsized":"2023-07-14T19:23:41.474300+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:03.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514703+0000","last_change":"2023-07-14T19:23:38.303732+0000","last_active":"2023-07-14T19:23:39.514703+0000","last_peered":"2023-07-14T19:23:39.514703+0000","last_clean":"2023-07-14T19:23:39.514703+0000","last_became_active":"2023-07-14T19:23:38.303296+0000","last_became_peered":"2023-07-14T19:23:38.303296+0000","last_unstale":"2023-07-14T19:23:39.514703+0000","last_undegraded":"2023-07-14T19:23:39.514703+0000","last_fullsized":"2023-07-14T19:23:39.514703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:55:39.991626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"80'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.475191+0000","last_change":"2023-07-14T19:23:37.461183+0000","last_active":"2023-07-14T19:23:41.475191+0000","last_peered":"2023-07-14T19:23:41.475191+0000","last_clean":"2023-07-14T19:23:41.475191+0000","last_became_active":"2023-07-14T19:23:37.460876+0000","last_became_peered":"2023-07-14T19:23:37.460876+0000","last_unstale":"2023-07-14T19:23:41.475191+0000","last_undegraded":"2023-07-14T19:23:41.475191+0000","last_fullsized":"2023-07-14T19:23:41.475191+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:31.746901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.286421+0000","last_change":"2023-07-14T19:23:38.292032+0000","last_active":"2023-07-14T19:23:39.286421+0000","last_peered":"2023-07-14T19:23:39.286421+0000","last_clean":"2023-07-14T19:23:39.286421+0000","last_became_active":"2023-07-14T19:23:38.291721+0000","last_became_peered":"2023-07-14T19:23:38.291721+0000","last_unstale":"2023-07-14T19:23:39.286421+0000","last_undegraded":"2023-07-14T19:23:39.286421+0000","last_fullsized":"2023-07-14T19:23:39.286421+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:58.334225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"80'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474819+0000","last_change":"2023-07-14T19:23:37.465597+0000","last_active":"2023-07-14T19:23:41.474819+0000","last_peered":"2023-07-14T19:23:41.474819+0000","last_clean":"2023-07-14T19:23:41.474819+0000","last_became_active":"2023-07-14T19:23:37.465233+0000","last_became_peered":"2023-07-14T19:23:37.465233+0000","last_unstale":"2023-07-14T19:23:41.474819+0000","last_undegraded":"2023-07-14T19:23:41.474819+0000","last_fullsized":"2023-07-14T19:23:41.474819+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:55.848783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472581+0000","last_change":"2023-07-14T19:23:38.295185+0000","last_active":"2023-07-14T19:23:39.472581+0000","last_peered":"2023-07-14T19:23:39.472581+0000","last_clean":"2023-07-14T19:23:39.472581+0000","last_became_active":"2023-07-14T19:23:38.294855+0000","last_became_peered":"2023-07-14T19:23:38.294855+0000","last_unstale":"2023-07-14T19:23:39.472581+0000","last_undegraded":"2023-07-14T19:23:39.472581+0000","last_fullsized":"2023-07-14T19:23:39.472581+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:48.045146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:40.262742+0000","last_change":"2023-07-14T19:23:37.463517+0000","last_active":"2023-07-14T19:23:40.262742+0000","last_peered":"2023-07-14T19:23:40.262742+0000","last_clean":"2023-07-14T19:23:40.262742+0000","last_became_active":"2023-07-14T19:23:37.463158+0000","last_became_peered":"2023-07-14T19:23:37.463158+0000","last_unstale":"2023-07-14T19:23:40.262742+0000","last_undegraded":"2023-07-14T19:23:40.262742+0000","last_fullsized":"2023-07-14T19:23:40.262742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:01.142000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283721+0000","last_change":"2023-07-14T19:23:38.305375+0000","last_active":"2023-07-14T19:23:39.283721+0000","last_peered":"2023-07-14T19:23:39.283721+0000","last_clean":"2023-07-14T19:23:39.283721+0000","last_became_active":"2023-07-14T19:23:38.305084+0000","last_became_peered":"2023-07-14T19:23:38.305084+0000","last_unstale":"2023-07-14T19:23:39.283721+0000","last_undegraded":"2023-07-14T19:23:39.283721+0000","last_fullsized":"2023-07-14T19:23:39.283721+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:08:51.652651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"80'4","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:41.474944+0000","last_change":"2023-07-14T19:23:37.458672+0000","last_active":"2023-07-14T19:23:41.474944+0000","last_peered":"2023-07-14T19:23:41.474944+0000","last_clean":"2023-07-14T19:23:41.474944+0000","last_became_active":"2023-07-14T19:23:37.458359+0000","last_became_peered":"2023-07-14T19:23:37.458359+0000","last_unstale":"2023-07-14T19:23:41.474944+0000","last_undegraded":"2023-07-14T19:23:41.474944+0000","last_fullsized":"2023-07-14T19:23:41.474944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:00:25.922910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515725+0000","last_change":"2023-07-14T19:23:38.301741+0000","last_active":"2023-07-14T19:23:39.515725+0000","last_peered":"2023-07-14T19:23:39.515725+0000","last_clean":"2023-07-14T19:23:39.515725+0000","last_became_active":"2023-07-14T19:23:38.301430+0000","last_became_peered":"2023-07-14T19:23:38.301430+0000","last_unstale":"2023-07-14T19:23:39.515725+0000","last_undegraded":"2023-07-14T19:23:39.515725+0000","last_fullsized":"2023-07-14T19:23:39.515725+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:13.088226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"80'3","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:42.296558+0000","last_change":"2023-07-14T19:23:37.461764+0000","last_active":"2023-07-14T19:23:42.296558+0000","last_peered":"2023-07-14T19:23:42.296558+0000","last_clean":"2023-07-14T19:23:42.296558+0000","last_became_active":"2023-07-14T19:23:37.460910+0000","last_became_peered":"2023-07-14T19:23:37.460910+0000","last_unstale":"2023-07-14T19:23:42.296558+0000","last_undegraded":"2023-07-14T19:23:42.296558+0000","last_fullsized":"2023-07-14T19:23:42.296558+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:39.478046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515703+0000","last_change":"2023-07-14T19:23:38.296214+0000","last_active":"2023-07-14T19:23:39.515703+0000","last_peered":"2023-07-14T19:23:39.515703+0000","last_clean":"2023-07-14T19:23:39.515703+0000","last_became_active":"2023-07-14T19:23:38.295148+0000","last_became_peered":"2023-07-14T19:23:38.295148+0000","last_unstale":"2023-07-14T19:23:39.515703+0000","last_undegraded":"2023-07-14T19:23:39.515703+0000","last_fullsized":"2023-07-14T19:23:39.515703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:30.117202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515939+0000","last_change":"2023-07-14T19:23:37.458245+0000","last_active":"2023-07-14T19:23:39.515939+0000","last_peered":"2023-07-14T19:23:39.515939+0000","last_clean":"2023-07-14T19:23:39.515939+0000","last_became_active":"2023-07-14T19:23:37.457784+0000","last_became_peered":"2023-07-14T19:23:37.457784+0000","last_unstale":"2023-07-14T19:23:39.515939+0000","last_undegraded":"2023-07-14T19:23:39.515939+0000","last_fullsized":"2023-07-14T19:23:39.515939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:47:00.389865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515370+0000","last_change":"2023-07-14T19:23:38.299823+0000","last_active":"2023-07-14T19:23:39.515370+0000","last_peered":"2023-07-14T19:23:39.515370+0000","last_clean":"2023-07-14T19:23:39.515370+0000","last_became_active":"2023-07-14T19:23:38.299496+0000","last_became_peered":"2023-07-14T19:23:38.299496+0000","last_unstale":"2023-07-14T19:23:39.515370+0000","last_undegraded":"2023-07-14T19:23:39.515370+0000","last_fullsized":"2023-07-14T19:23:39.515370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:23:40.353266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514282+0000","last_change":"2023-07-14T19:23:37.463439+0000","last_active":"2023-07-14T19:23:39.514282+0000","last_peered":"2023-07-14T19:23:39.514282+0000","last_clean":"2023-07-14T19:23:39.514282+0000","last_became_active":"2023-07-14T19:23:37.463100+0000","last_became_peered":"2023-07-14T19:23:37.463100+0000","last_unstale":"2023-07-14T19:23:39.514282+0000","last_undegraded":"2023-07-14T19:23:39.514282+0000","last_fullsized":"2023-07-14T19:23:39.514282+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:36.440729+0000","last_clean_scrub_stamp":"2023-07-14T19:23:36.440729+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:56.140535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515366+0000","last_change":"2023-07-14T19:23:38.296265+0000","last_active":"2023-07-14T19:23:39.515366+0000","last_peered":"2023-07-14T19:23:39.515366+0000","last_clean":"2023-07-14T19:23:39.515366+0000","last_became_active":"2023-07-14T19:23:38.295918+0000","last_became_peered":"2023-07-14T19:23:38.295918+0000","last_unstale":"2023-07-14T19:23:39.515366+0000","last_undegraded":"2023-07-14T19:23:39.515366+0000","last_fullsized":"2023-07-14T19:23:39.515366+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:38.769993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472522+0000","last_change":"2023-07-14T19:23:38.300200+0000","last_active":"2023-07-14T19:23:39.472522+0000","last_peered":"2023-07-14T19:23:39.472522+0000","last_clean":"2023-07-14T19:23:39.472522+0000","last_became_active":"2023-07-14T19:23:38.299979+0000","last_became_peered":"2023-07-14T19:23:38.299979+0000","last_unstale":"2023-07-14T19:23:39.472522+0000","last_undegraded":"2023-07-14T19:23:39.472522+0000","last_fullsized":"2023-07-14T19:23:39.472522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:00.789219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514842+0000","last_change":"2023-07-14T19:23:38.300696+0000","last_active":"2023-07-14T19:23:39.514842+0000","last_peered":"2023-07-14T19:23:39.514842+0000","last_clean":"2023-07-14T19:23:39.514842+0000","last_became_active":"2023-07-14T19:23:38.300377+0000","last_became_peered":"2023-07-14T19:23:38.300377+0000","last_unstale":"2023-07-14T19:23:39.514842+0000","last_undegraded":"2023-07-14T19:23:39.514842+0000","last_fullsized":"2023-07-14T19:23:39.514842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:39:05.440224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514814+0000","last_change":"2023-07-14T19:23:38.298616+0000","last_active":"2023-07-14T19:23:39.514814+0000","last_peered":"2023-07-14T19:23:39.514814+0000","last_clean":"2023-07-14T19:23:39.514814+0000","last_became_active":"2023-07-14T19:23:38.298313+0000","last_became_peered":"2023-07-14T19:23:38.298313+0000","last_unstale":"2023-07-14T19:23:39.514814+0000","last_undegraded":"2023-07-14T19:23:39.514814+0000","last_fullsized":"2023-07-14T19:23:39.514814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:46:49.950796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514982+0000","last_change":"2023-07-14T19:23:38.297185+0000","last_active":"2023-07-14T19:23:39.514982+0000","last_peered":"2023-07-14T19:23:39.514982+0000","last_clean":"2023-07-14T19:23:39.514982+0000","last_became_active":"2023-07-14T19:23:38.296883+0000","last_became_peered":"2023-07-14T19:23:38.296883+0000","last_unstale":"2023-07-14T19:23:39.514982+0000","last_undegraded":"2023-07-14T19:23:39.514982+0000","last_fullsized":"2023-07-14T19:23:39.514982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:08:27.715078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.514907+0000","last_change":"2023-07-14T19:23:38.296636+0000","last_active":"2023-07-14T19:23:39.514907+0000","last_peered":"2023-07-14T19:23:39.514907+0000","last_clean":"2023-07-14T19:23:39.514907+0000","last_became_active":"2023-07-14T19:23:38.296289+0000","last_became_peered":"2023-07-14T19:23:38.296289+0000","last_unstale":"2023-07-14T19:23:39.514907+0000","last_undegraded":"2023-07-14T19:23:39.514907+0000","last_fullsized":"2023-07-14T19:23:39.514907+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:18:30.937689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472626+0000","last_change":"2023-07-14T19:23:38.300340+0000","last_active":"2023-07-14T19:23:39.472626+0000","last_peered":"2023-07-14T19:23:39.472626+0000","last_clean":"2023-07-14T19:23:39.472626+0000","last_became_active":"2023-07-14T19:23:38.299520+0000","last_became_peered":"2023-07-14T19:23:38.299520+0000","last_unstale":"2023-07-14T19:23:39.472626+0000","last_undegraded":"2023-07-14T19:23:39.472626+0000","last_fullsized":"2023-07-14T19:23:39.472626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:03:47.657130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515538+0000","last_change":"2023-07-14T19:23:38.294136+0000","last_active":"2023-07-14T19:23:39.515538+0000","last_peered":"2023-07-14T19:23:39.515538+0000","last_clean":"2023-07-14T19:23:39.515538+0000","last_became_active":"2023-07-14T19:23:38.293785+0000","last_became_peered":"2023-07-14T19:23:38.293785+0000","last_unstale":"2023-07-14T19:23:39.515538+0000","last_undegraded":"2023-07-14T19:23:39.515538+0000","last_fullsized":"2023-07-14T19:23:39.515538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:21.794486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.284002+0000","last_change":"2023-07-14T19:23:38.305648+0000","last_active":"2023-07-14T19:23:39.284002+0000","last_peered":"2023-07-14T19:23:39.284002+0000","last_clean":"2023-07-14T19:23:39.284002+0000","last_became_active":"2023-07-14T19:23:38.305314+0000","last_became_peered":"2023-07-14T19:23:38.305314+0000","last_unstale":"2023-07-14T19:23:39.284002+0000","last_undegraded":"2023-07-14T19:23:39.284002+0000","last_fullsized":"2023-07-14T19:23:39.284002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:01:05.266523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515131+0000","last_change":"2023-07-14T19:23:38.298189+0000","last_active":"2023-07-14T19:23:39.515131+0000","last_peered":"2023-07-14T19:23:39.515131+0000","last_clean":"2023-07-14T19:23:39.515131+0000","last_became_active":"2023-07-14T19:23:38.297781+0000","last_became_peered":"2023-07-14T19:23:38.297781+0000","last_unstale":"2023-07-14T19:23:39.515131+0000","last_undegraded":"2023-07-14T19:23:39.515131+0000","last_fullsized":"2023-07-14T19:23:39.515131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:49:59.706932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.283450+0000","last_change":"2023-07-14T19:23:38.293567+0000","last_active":"2023-07-14T19:23:39.283450+0000","last_peered":"2023-07-14T19:23:39.283450+0000","last_clean":"2023-07-14T19:23:39.283450+0000","last_became_active":"2023-07-14T19:23:38.293222+0000","last_became_peered":"2023-07-14T19:23:38.293222+0000","last_unstale":"2023-07-14T19:23:39.283450+0000","last_undegraded":"2023-07-14T19:23:39.283450+0000","last_fullsized":"2023-07-14T19:23:39.283450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:26:28.545521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515396+0000","last_change":"2023-07-14T19:23:38.294168+0000","last_active":"2023-07-14T19:23:39.515396+0000","last_peered":"2023-07-14T19:23:39.515396+0000","last_clean":"2023-07-14T19:23:39.515396+0000","last_became_active":"2023-07-14T19:23:38.293665+0000","last_became_peered":"2023-07-14T19:23:38.293665+0000","last_unstale":"2023-07-14T19:23:39.515396+0000","last_undegraded":"2023-07-14T19:23:39.515396+0000","last_fullsized":"2023-07-14T19:23:39.515396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:49.466175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.515479+0000","last_change":"2023-07-14T19:23:38.296535+0000","last_active":"2023-07-14T19:23:39.515479+0000","last_peered":"2023-07-14T19:23:39.515479+0000","last_clean":"2023-07-14T19:23:39.515479+0000","last_became_active":"2023-07-14T19:23:38.296189+0000","last_became_peered":"2023-07-14T19:23:38.296189+0000","last_unstale":"2023-07-14T19:23:39.515479+0000","last_undegraded":"2023-07-14T19:23:39.515479+0000","last_fullsized":"2023-07-14T19:23:39.515479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:36:54.183312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2023-07-14T19:23:39.472559+0000","last_change":"2023-07-14T19:23:38.296867+0000","last_active":"2023-07-14T19:23:39.472559+0000","last_peered":"2023-07-14T19:23:39.472559+0000","last_clean":"2023-07-14T19:23:39.472559+0000","last_became_active":"2023-07-14T19:23:38.296560+0000","last_became_peered":"2023-07-14T19:23:38.296560+0000","last_unstale":"2023-07-14T19:23:39.472559+0000","last_undegraded":"2023-07-14T19:23:39.472559+0000","last_fullsized":"2023-07-14T19:23:39.472559+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:23:37.442666+0000","last_clean_scrub_stamp":"2023-07-14T19:23:37.442666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:48:16.893808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62431,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":276,"num_read_kb":206,"num_write":149,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":187293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547221,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":414159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.432}]},{"osd":1,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.55100000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.67700000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.74299999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.755},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.72499999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.80100000000000005}]}]},{"osd":10,"up_from":68,"seq":292057776152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1063794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.373},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.66900000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.503}]},{"osd":4,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.53300000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.72799999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.54600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.74199999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.70399999999999996}]},{"osd":11,"last update":"Fri Jul 14 19:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.77900000000000003}]}]},{"osd":9,"up_from":63,"seq":270582939676,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":880640,"data_stored":414395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.30199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.784,"5min":1.784,"15min":1.784},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.54400000000000004,"15min":0.56999999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.73799999999999999,"5min":2.6080000000000001,"15min":2.6080000000000001},"last":0.31900000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64000000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.42499999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.80600000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.73,"5min":2.657,"15min":2.657},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.50900000000000001,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":0.434}]},{"osd":4,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67400000000000004,"15min":0.67500000000000004},"min":{"1min":0.48499999999999999,"5min":0.443,"15min":0.443},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.48799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.696,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":0.46700000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.76500000000000001,"15min":0.79900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.745,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.67500000000000004,"15min":0.68700000000000006},"min":{"1min":0.51600000000000001,"5min":0.499,"15min":0.499},"max":{"1min":0.83099999999999996,"5min":1.05,"15min":1.05},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.70199999999999996,"15min":0.71999999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.835,"5min":2.6699999999999999,"15min":2.6699999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.81399999999999995,"15min":0.84199999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.8560000000000001,"5min":3.9740000000000002,"15min":3.9740000000000002},"last":0.59999999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.83599999999999997,"15min":0.86099999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.77400000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.73899999999999999,"15min":0.753},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.88200000000000001,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.504}]},{"osd":8,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.76600000000000001,"15min":0.77600000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.83399999999999996,"15min":0.86399999999999999},"min":{"1min":0.53800000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.78400000000000003,"5min":2.7309999999999999,"15min":2.7309999999999999},"last":0.66200000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.79,"5min":2.79,"15min":2.79},"last":0.628}]}]},{"osd":8,"up_from":56,"seq":240518168609,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":413984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.504,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78800000000000003,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.745}]},{"osd":1,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.82999999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.28399999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.621},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.503,"15min":0.505},"min":{"1min":0.29899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.64000000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59899999999999998,"15min":0.59999999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76400000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59799999999999998,"15min":0.60599999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.67200000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84099999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.39700000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83499999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.81000000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.39300000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.88300000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.89500000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66900000000000004,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67200000000000004,"15min":0.67300000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.879}]},{"osd":7,"last update":"Fri Jul 14 19:23:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.50700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.73399999999999999,"15min":0.73099999999999998},"min":{"1min":0.56399999999999995,"5min":0.443,"15min":0.443},"max":{"1min":0.999,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.93600000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.68400000000000005,"15min":0.66900000000000004},"min":{"1min":0.59199999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.70999999999999996,"15min":0.69799999999999995},"min":{"1min":0.56200000000000006,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.95199999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.97899999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]}]},{"osd":7,"up_from":48,"seq":206158430244,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28456,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716696,"statfs":{"total":95995035648,"available":95965896704,"internally_reserved":0,"allocated":1474560,"data_stored":1006154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73099999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.82299999999999995,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.43099999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42499999999999999,"15min":0.42099999999999999},"min":{"1min":0.32900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.42399999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78600000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.45800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.68899999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.60099999999999998,"15min":0.60199999999999998},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.58199999999999996}]},{"osd":4,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.502,"15min":0.49399999999999999},"min":{"1min":0.36299999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76600000000000001,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.441}]},{"osd":5,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.621,"15min":0.624},"min":{"1min":0.47699999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66600000000000004,"15min":0.67000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.85799999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.51300000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66400000000000003},"min":{"1min":0.41199999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:23:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.68999999999999995,"15min":0.68500000000000005},"min":{"1min":0.49199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.67100000000000004,"15min":0.66600000000000004},"min":{"1min":0.48199999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.75800000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:24:18 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.58299999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.73399999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.83299999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]}]},{"osd":6,"up_from":43,"seq":184683593770,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":413790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.48799999999999999,"15min":0.432},"min":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":11.576000000000001,"5min":11.576000000000001,"15min":11.576000000000001},"last":0.497},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.439,"15min":0.443},"min":{"1min":0.29499999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.33600000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.61499999999999999,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":11.202,"5min":11.202,"15min":11.202},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.999,"5min":0.66700000000000004,"15min":0.60199999999999998},"min":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":11.505000000000001,"5min":11.505000000000001,"15min":11.505000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.61399999999999999,"15min":0.60799999999999998},"min":{"1min":0.45200000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82199999999999995,"5min":1.071,"15min":1.071},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.71399999999999997,"15min":0.64600000000000002},"min":{"1min":0.435,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":11.390000000000001,"5min":11.390000000000001,"15min":11.390000000000001},"last":0.54800000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.58999999999999997,"15min":0.53600000000000003},"min":{"1min":0.34399999999999997,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":11.539999999999999,"5min":11.539999999999999,"15min":11.539999999999999},"last":0.375},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.44400000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.65400000000000003,"15min":0.59499999999999997},"min":{"1min":0.39800000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":11.680999999999999,"5min":11.680999999999999,"15min":11.680999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.022,"5min":0.73899999999999999,"15min":0.67300000000000004},"min":{"1min":0.47699999999999998,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":11.442,"5min":11.442,"15min":11.442},"last":0.60699999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.67100000000000004,"15min":0.66000000000000003},"min":{"1min":0.441,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86799999999999999,"5min":1.107,"15min":1.107},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.69099999999999995,"15min":0.63700000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":11.747999999999999,"5min":11.747999999999999,"15min":11.747999999999999},"last":0.46800000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.74299999999999999,"15min":0.73999999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.59599999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:23:56 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":0.79400000000000004,"15min":0.73799999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":11.778,"5min":11.778,"15min":11.778},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.21,"5min":0.875,"15min":0.81899999999999995},"min":{"1min":0.53800000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":11.929,"5min":11.929,"15min":11.929},"last":0.69199999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:24:17 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.84699999999999998,"15min":0.81399999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":11.888999999999999,"5min":11.888999999999999,"15min":11.888999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.77900000000000003,"15min":0.80400000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89100000000000001,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.54000000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.625},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.67200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":471831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.53200000000000003,"15min":0.52200000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83999999999999997,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.83199999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.54300000000000004},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.56699999999999995,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64700000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.51400000000000001,"15min":0.504},"min":{"1min":0.42399999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.35599999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69899999999999995,"5min":1.028,"15min":1.028},"last":0.60299999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.40300000000000002},"min":{"1min":0.27600000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.443},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.433,"15min":0.42499999999999999},"min":{"1min":0.32200000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.41199999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.61199999999999999,"15min":0.58199999999999996},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.71399999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:24:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.63200000000000001,"15min":0.61099999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.56999999999999995,"15min":0.55400000000000005},"min":{"1min":0.51000000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.53100000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54100000000000004,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.68500000000000005,"15min":0.67700000000000005},"min":{"1min":0.46600000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.71699999999999997,"15min":0.70499999999999996},"min":{"1min":0.52400000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.0269999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.755}]},{"osd":9,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.75800000000000001,"15min":0.746},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.042,"5min":1.079,"15min":1.079},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.68500000000000005,"15min":0.67500000000000004},"min":{"1min":0.59699999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73999999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:23:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.879}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":415055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.52500000000000002,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.84599999999999997,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.51400000000000001,"15min":0.46800000000000003},"min":{"1min":0.438,"5min":0.318,"15min":0.318},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.51400000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52000000000000002,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.65400000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54800000000000004,"15min":0.52500000000000002},"min":{"1min":0.371,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79400000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.71399999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:22:16 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.432,"15min":0.42599999999999999},"min":{"1min":0.26300000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.433},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.62,"15min":0.60799999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.623},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59099999999999997,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.314,"15min":0.314},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.54700000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.45900000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78600000000000003,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60899999999999999,"15min":0.60399999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76300000000000001,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.748}]},{"osd":6,"last update":"Fri Jul 14 19:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.60099999999999998,"15min":0.59499999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:23:34 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.67600000000000005,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.434,"15min":0.434},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.73099999999999998,"15min":0.72699999999999998},"min":{"1min":0.55900000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.58599999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.70899999999999996,"15min":0.72399999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.90600000000000003,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.73599999999999999,"15min":0.746},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.997,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.73199999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.56299999999999994}]},{"osd":10,"last update":"Fri Jul 14 19:23:39 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":1.028},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.66000000000000003}]},{"osd":11,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.69999999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":473796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52700000000000002,"15min":0.51300000000000001},"min":{"1min":0.41399999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.505,"15min":0.49199999999999999},"min":{"1min":0.35999999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.72799999999999998,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53200000000000003,"15min":0.52100000000000002},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78600000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.55100000000000005,"15min":0.50900000000000001},"min":{"1min":0.45200000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79100000000000004,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.67800000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57599999999999996,"15min":0.55400000000000005},"min":{"1min":0.36899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82899999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.623,"15min":0.57899999999999996},"min":{"1min":0.379,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.056,"5min":5.056,"15min":5.056},"last":0.56399999999999995}]},{"osd":4,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.50700000000000001},"min":{"1min":0.438,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72599999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69299999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.44600000000000001,"15min":0.42999999999999999},"min":{"1min":0.309,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63600000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.46800000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.63200000000000001,"15min":0.61599999999999999},"min":{"1min":0.42199999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.434},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.59399999999999997,"15min":0.57599999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.41799999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.434,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82799999999999996,"5min":3.6579999999999999,"15min":3.6579999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.79900000000000004,"15min":0.82899999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.79500000000000004,"5min":3.8769999999999998,"15min":3.8769999999999998},"last":0.45900000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:23:58 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69399999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68000000000000005,"15min":0.68600000000000005},"min":{"1min":0.40100000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.85199999999999998,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.52700000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:24:16 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.745,"15min":0.746},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.623},{"interface":"front","average":{"1min":0.75,"5min":0.70399999999999996,"15min":0.69599999999999995},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.61099999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.78600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":3,"up_from":24,"seq":103079215157,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":414107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.36799999999999999,"15min":0.34899999999999998},"min":{"1min":0.30399999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.39000000000000001,"15min":0.372},"min":{"1min":0.29399999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.377}]},{"osd":1,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.54200000000000004,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.53600000000000003,"15min":0.53200000000000003},"min":{"1min":0.436,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.48999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.57899999999999996,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.46300000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.57699999999999996,"15min":0.55500000000000005},"min":{"1min":0.47299999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.625,"15min":0.59999999999999998},"min":{"1min":0.41499999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.67200000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.625,"15min":0.60099999999999998},"min":{"1min":0.48699999999999999,"5min":0.37,"15min":0.37},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.56200000000000006,"15min":0.54000000000000004},"min":{"1min":0.36299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86599999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.35099999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.38900000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.503}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.61499999999999999,"15min":0.59799999999999998},"min":{"1min":0.443,"5min":0.436,"15min":0.436},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.73099999999999998,"15min":0.71699999999999997},"min":{"1min":0.54300000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.754}]},{"osd":8,"last update":"Fri Jul 14 19:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.54800000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.623}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.84799999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":414173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.34499999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.44,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.73999999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.46200000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.36799999999999999,"15min":0.33800000000000002},"min":{"1min":0.29499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.78200000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.39700000000000002,"15min":0.377},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.81599999999999995,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.33400000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.55300000000000005,"15min":0.51900000000000002},"min":{"1min":0.48799999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88900000000000001,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52600000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.40000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84799999999999998,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.5},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53700000000000003,"15min":0.52400000000000002},"min":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.54800000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59799999999999998,"15min":0.57799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.878,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.621,"15min":0.58099999999999996},"min":{"1min":0.55600000000000005,"5min":0.372,"15min":0.372},"max":{"1min":0.96099999999999997,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.57199999999999995,"15min":0.56899999999999995},"min":{"1min":0.42299999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.36899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.94799999999999995,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.56799999999999995}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.53900000000000003,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.442,"5min":0.377,"15min":0.377},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.78500000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70999999999999996,"15min":0.70499999999999996},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.69099999999999995,"15min":0.68700000000000006},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.92500000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:23:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.64300000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":1.0169999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]}]},{"osd":5,"up_from":36,"seq":154618822701,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28464,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716688,"statfs":{"total":95995035648,"available":95965888512,"internally_reserved":0,"allocated":1482752,"data_stored":1005555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53400000000000003,"15min":0.51200000000000001},"min":{"1min":0.32900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8049999999999999},"last":0.441},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.53900000000000003,"15min":0.53400000000000003},"min":{"1min":0.35899999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53400000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.53700000000000003,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.40200000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.67900000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40100000000000002,"15min":0.39300000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56799999999999995,"15min":0.55700000000000005},"min":{"1min":0.35399999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77500000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57499999999999996,"15min":0.56399999999999995},"min":{"1min":0.33500000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77200000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.55800000000000005,"15min":0.54400000000000004},"min":{"1min":0.48599999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55600000000000005,"15min":0.53800000000000003},"min":{"1min":0.41199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63300000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.629,"15min":0.60899999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.65000000000000002,"15min":0.625},"min":{"1min":0.53100000000000003,"5min":0.37,"15min":0.37},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.60099999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:23:35 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63800000000000001,"15min":0.63300000000000001},"min":{"1min":0.51300000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.64600000000000002,"15min":0.64400000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.77900000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:23:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.54600000000000004,"15min":0.53500000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.59199999999999997,"15min":0.57899999999999996},"min":{"1min":0.38600000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.42399999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:24:19 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.72599999999999998,"15min":0.70699999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.71699999999999997,"15min":0.70999999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.65100000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:24:26.717 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-14T19:24:26.717 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-14T19:24:26.718 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-14T19:24:26.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph health --format=json 2023-07-14T19:24:26.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:26 smithi070 ceph-mon[98015]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:24:26.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:26 smithi104 ceph-mon[98435]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:24:27.277 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:28.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.466+0000 7f4d7cdea700 1 -- 172.21.15.53:0/4223692552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78073270 msgr2=0x7f4d7810c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:28.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.466+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/4223692552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78073270 0x7f4d7810c060 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009990 tx=0x7f4d60009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.466+0000 7f4d7cdea700 1 -- 172.21.15.53:0/4223692552 shutdown_connections 2023-07-14T19:24:28.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.466+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/4223692552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d7810c5a0 0x7f4d7810e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.466+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/4223692552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d78072960 0x7f4d78072d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.466+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/4223692552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78073270 0x7f4d7810c060 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.466+0000 7f4d7cdea700 1 -- 172.21.15.53:0/4223692552 >> 172.21.15.53:0/4223692552 conn(0x7f4d780fb7c0 msgr2=0x7f4d780fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:28.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.466+0000 7f4d7cdea700 1 -- 172.21.15.53:0/4223692552 shutdown_connections 2023-07-14T19:24:28.469 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.466+0000 7f4d7cdea700 1 -- 172.21.15.53:0/4223692552 wait complete. 2023-07-14T19:24:28.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.467+0000 7f4d7cdea700 1 Processor -- start 2023-07-14T19:24:28.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.467+0000 7f4d7cdea700 1 -- start start 2023-07-14T19:24:28.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.467+0000 7f4d7cdea700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78072960 0x7f4d78197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:28.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.467+0000 7f4d7cdea700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d78073270 0x7f4d781983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:28.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.467+0000 7f4d7cdea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d7810c5a0 0x7f4d7819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:28.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.467+0000 7f4d7cdea700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7819ce70 con 0x7f4d78072960 2023-07-14T19:24:28.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.467+0000 7f4d7cdea700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7819cfe0 con 0x7f4d78073270 2023-07-14T19:24:28.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.467+0000 7f4d7cdea700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7819d1a0 con 0x7f4d7810c5a0 2023-07-14T19:24:28.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78072960 0x7f4d78197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:28.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d76d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d7810c5a0 0x7f4d7819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:28.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7659c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78072960 0x7f4d78197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50842/0 (socket says 172.21.15.53:50842) 2023-07-14T19:24:28.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7659c700 1 -- 172.21.15.53:0/2732978535 learned_addr learned my addr 172.21.15.53:0/2732978535 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:28.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d75d9b700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d78073270 0x7f4d781983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:28.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7659c700 1 -- 172.21.15.53:0/2732978535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d7810c5a0 msgr2=0x7f4d7819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:28.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7659c700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d7810c5a0 0x7f4d7819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7659c700 1 -- 172.21.15.53:0/2732978535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d78073270 msgr2=0x7f4d781983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:28.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7659c700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d78073270 0x7f4d781983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7659c700 1 -- 172.21.15.53:0/2732978535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600095f0 con 0x7f4d78072960 2023-07-14T19:24:28.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7659c700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78072960 0x7f4d78197eb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800c920 tx=0x7f4d6800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:28.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d6f7fe700 1 -- 172.21.15.53:0/2732978535 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d68004a30 con 0x7f4d78072960 2023-07-14T19:24:28.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d6f7fe700 1 -- 172.21.15.53:0/2732978535 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d680054e0 con 0x7f4d78072960 2023-07-14T19:24:28.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7819d480 con 0x7f4d78072960 2023-07-14T19:24:28.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d6f7fe700 1 -- 172.21.15.53:0/2732978535 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d68007bd0 con 0x7f4d78072960 2023-07-14T19:24:28.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.468+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d781a4ce0 con 0x7f4d78072960 2023-07-14T19:24:28.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.469+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d78106ec0 con 0x7f4d78072960 2023-07-14T19:24:28.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.471+0000 7f4d6f7fe700 1 -- 172.21.15.53:0/2732978535 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4d68007d50 con 0x7f4d78072960 2023-07-14T19:24:28.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.472+0000 7f4d6f7fe700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d640a1260 0x7f4d640a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:28.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.472+0000 7f4d75d9b700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d640a1260 0x7f4d640a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:28.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.472+0000 7f4d6f7fe700 1 -- 172.21.15.53:0/2732978535 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4d680cc7e0 con 0x7f4d78072960 2023-07-14T19:24:28.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.472+0000 7f4d75d9b700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d640a1260 0x7f4d640a3710 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000b5c0 tx=0x7f4d60005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:28.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.475+0000 7f4d6f7fe700 1 -- 172.21.15.53:0/2732978535 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4d68096000 con 0x7f4d78072960 2023-07-14T19:24:28.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:28 smithi053 ceph-mon[96565]: pgmap v273: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:24:28.718 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.716+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4d7804e340 con 0x7f4d78072960 2023-07-14T19:24:28.718 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.717+0000 7f4d6f7fe700 1 -- 172.21.15.53:0/2732978535 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4d6801d020 con 0x7f4d78072960 2023-07-14T19:24:28.718 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:28.718 INFO:teuthology.orchestra.run.smithi053.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-14T19:24:28.720 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.718+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d640a1260 msgr2=0x7f4d640a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:28.720 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.718+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d640a1260 0x7f4d640a3710 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4d6000b5c0 tx=0x7f4d60005960 comp rx=0 tx=0).stop 2023-07-14T19:24:28.720 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.718+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78072960 msgr2=0x7f4d78197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:28.720 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.718+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78072960 0x7f4d78197eb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800c920 tx=0x7f4d6800cc30 comp rx=0 tx=0).stop 2023-07-14T19:24:28.720 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.718+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 shutdown_connections 2023-07-14T19:24:28.720 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.718+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4d640a1260 0x7f4d640a3710 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.721 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.718+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d7810c5a0 0x7f4d7819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.721 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.719+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d78072960 0x7f4d78197eb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.721 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.719+0000 7f4d7cdea700 1 --2- 172.21.15.53:0/2732978535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d78073270 0x7f4d781983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:28.721 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.719+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 >> 172.21.15.53:0/2732978535 conn(0x7f4d780fb7c0 msgr2=0x7f4d78104000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:28.721 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.719+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 shutdown_connections 2023-07-14T19:24:28.721 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:28.719+0000 7f4d7cdea700 1 -- 172.21.15.53:0/2732978535 wait complete. 2023-07-14T19:24:28.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:28 smithi070 ceph-mon[98015]: pgmap v273: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:24:28.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:28 smithi104 ceph-mon[98435]: pgmap v273: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:24:29.139 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-14T19:24:29.140 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:24:29.269 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 -- 172.21.15.53:0/2893688927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77000ffcf0 msgr2=0x7f7700100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.270 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 --2- 172.21.15.53:0/2893688927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77000ffcf0 0x7f7700100140 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009990 tx=0x7f76e8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.270 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 -- 172.21.15.53:0/2893688927 shutdown_connections 2023-07-14T19:24:29.270 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 --2- 172.21.15.53:0/2893688927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7700100680 0x7f7700106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.270 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 --2- 172.21.15.53:0/2893688927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77000feb00 0x7f77000fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.270 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 --2- 172.21.15.53:0/2893688927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77000ffcf0 0x7f7700100140 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 -- 172.21.15.53:0/2893688927 >> 172.21.15.53:0/2893688927 conn(0x7f77000fa100 msgr2=0x7f77000fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:29.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 -- 172.21.15.53:0/2893688927 shutdown_connections 2023-07-14T19:24:29.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 -- 172.21.15.53:0/2893688927 wait complete. 2023-07-14T19:24:29.271 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.268+0000 7f770512b700 1 Processor -- start 2023-07-14T19:24:29.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f770512b700 1 -- start start 2023-07-14T19:24:29.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f770512b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77000feb00 0x7f7700068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f770512b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77000ffcf0 0x7f770019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.273 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f770512b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7700100680 0x7f770019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f770512b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770019d420 con 0x7f7700100680 2023-07-14T19:24:29.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f770512b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770019d590 con 0x7f77000feb00 2023-07-14T19:24:29.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f770512b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770019d750 con 0x7f77000ffcf0 2023-07-14T19:24:29.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76ff59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7700100680 0x7f770019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76ff59e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7700100680 0x7f770019cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50858/0 (socket says 172.21.15.53:50858) 2023-07-14T19:24:29.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76ff59e700 1 -- 172.21.15.53:0/1468782509 learned_addr learned my addr 172.21.15.53:0/1468782509 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:29.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76fed9d700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77000feb00 0x7f7700068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76fe59c700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77000ffcf0 0x7f770019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76ff59e700 1 -- 172.21.15.53:0/1468782509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77000ffcf0 msgr2=0x7f770019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76ff59e700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77000ffcf0 0x7f770019a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76ff59e700 1 -- 172.21.15.53:0/1468782509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77000feb00 msgr2=0x7f7700068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76ff59e700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77000feb00 0x7f7700068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76ff59e700 1 -- 172.21.15.53:0/1468782509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e80095f0 con 0x7f7700100680 2023-07-14T19:24:29.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76fe59c700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77000ffcf0 0x7f770019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:29.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.269+0000 7f76fed9d700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77000feb00 0x7f7700068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:29.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.270+0000 7f76ff59e700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7700100680 0x7f770019cc80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f76f800d980 tx=0x7f76f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:29.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.270+0000 7f76f7fff700 1 -- 172.21.15.53:0/1468782509 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8004a30 con 0x7f7700100680 2023-07-14T19:24:29.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.270+0000 7f76f7fff700 1 -- 172.21.15.53:0/1468782509 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76f800b5c0 con 0x7f7700100680 2023-07-14T19:24:29.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.270+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770019da30 con 0x7f7700100680 2023-07-14T19:24:29.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.270+0000 7f76f7fff700 1 -- 172.21.15.53:0/1468782509 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8009b50 con 0x7f7700100680 2023-07-14T19:24:29.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.270+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770019df40 con 0x7f7700100680 2023-07-14T19:24:29.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.271+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7700106030 con 0x7f7700100680 2023-07-14T19:24:29.278 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.272+0000 7f76f7fff700 1 -- 172.21.15.53:0/1468782509 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f76f8009cb0 con 0x7f7700100680 2023-07-14T19:24:29.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.273+0000 7f76f7fff700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f76ec0a12a0 0x7f76ec0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.273+0000 7f76fed9d700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f76ec0a12a0 0x7f76ec0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.273+0000 7f76f7fff700 1 -- 172.21.15.53:0/1468782509 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f76f80cc800 con 0x7f7700100680 2023-07-14T19:24:29.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.273+0000 7f76fed9d700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f76ec0a12a0 0x7f76ec0a3750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f77000ffb50 tx=0x7f76f000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:29.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.277+0000 7f76f7fff700 1 -- 172.21.15.53:0/1468782509 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f76f8095ea0 con 0x7f7700100680 2023-07-14T19:24:29.480 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.479+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f770002ca90 con 0x7f7700100680 2023-07-14T19:24:29.481 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.480+0000 7f76f7fff700 1 -- 172.21.15.53:0/1468782509 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22054 (secure 0 0 0) 0x7f76f8017090 con 0x7f7700100680 2023-07-14T19:24:29.482 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:29.482 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":80,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:23:39.280266+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:24:29.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f76ec0a12a0 msgr2=0x7f76ec0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f76ec0a12a0 0x7f76ec0a3750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f77000ffb50 tx=0x7f76f000b040 comp rx=0 tx=0).stop 2023-07-14T19:24:29.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7700100680 msgr2=0x7f770019cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7700100680 0x7f770019cc80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f76f800d980 tx=0x7f76f800dc90 comp rx=0 tx=0).stop 2023-07-14T19:24:29.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 shutdown_connections 2023-07-14T19:24:29.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f76ec0a12a0 0x7f76ec0a3750 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7700100680 0x7f770019cc80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77000feb00 0x7f7700068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 --2- 172.21.15.53:0/1468782509 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f77000ffcf0 0x7f770019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 >> 172.21.15.53:0/1468782509 conn(0x7f77000fa100 msgr2=0x7f7700103170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:29.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 shutdown_connections 2023-07-14T19:24:29.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.482+0000 7f770512b700 1 -- 172.21.15.53:0/1468782509 wait complete. 2023-07-14T19:24:29.495 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:29 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2732978535' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:24:29.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:19:51.775369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:23:35.935538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:23:37.252372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:24:29.497 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:24:29.625 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 -- 172.21.15.53:0/798439649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4754100650 msgr2=0x7f4754106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.625 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 --2- 172.21.15.53:0/798439649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4754100650 0x7f4754106f10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f474c00b0a0 tx=0x7f474c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 -- 172.21.15.53:0/798439649 shutdown_connections 2023-07-14T19:24:29.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 --2- 172.21.15.53:0/798439649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4754100650 0x7f4754106f10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 --2- 172.21.15.53:0/798439649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47540fead0 0x7f47540feee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 --2- 172.21.15.53:0/798439649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47540ffcc0 0x7f4754100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 -- 172.21.15.53:0/798439649 >> 172.21.15.53:0/798439649 conn(0x7f47540fa0d0 msgr2=0x7f47540fc520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:29.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 -- 172.21.15.53:0/798439649 shutdown_connections 2023-07-14T19:24:29.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 -- 172.21.15.53:0/798439649 wait complete. 2023-07-14T19:24:29.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 Processor -- start 2023-07-14T19:24:29.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.624+0000 7f475b7fb700 1 -- start start 2023-07-14T19:24:29.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f475b7fb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47540fead0 0x7f4754068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f475b7fb700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47540ffcc0 0x7f475419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f475b7fb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4754100650 0x7f475419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f475b7fb700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475419d530 con 0x7f47540ffcc0 2023-07-14T19:24:29.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f475b7fb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475419d6a0 con 0x7f47540fead0 2023-07-14T19:24:29.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f475b7fb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475419d860 con 0x7f4754100650 2023-07-14T19:24:29.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4758d96700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47540ffcc0 0x7f475419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4758d96700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47540ffcc0 0x7f475419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50870/0 (socket says 172.21.15.53:50870) 2023-07-14T19:24:29.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4758d96700 1 -- 172.21.15.53:0/1460033378 learned_addr learned my addr 172.21.15.53:0/1460033378 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:29.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4759d98700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4754100650 0x7f475419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4759597700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47540fead0 0x7f4754068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4758d96700 1 -- 172.21.15.53:0/1460033378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4754100650 msgr2=0x7f475419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4758d96700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4754100650 0x7f475419cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4758d96700 1 -- 172.21.15.53:0/1460033378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47540fead0 msgr2=0x7f4754068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4758d96700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47540fead0 0x7f4754068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4758d96700 1 -- 172.21.15.53:0/1460033378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c009d00 con 0x7f47540ffcc0 2023-07-14T19:24:29.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4759597700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47540fead0 0x7f4754068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:29.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f4758d96700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47540ffcc0 0x7f475419a750 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f474400c920 tx=0x7f474400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:29.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f474a7fc700 1 -- 172.21.15.53:0/1460033378 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744007b00 con 0x7f47540ffcc0 2023-07-14T19:24:29.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f474a7fc700 1 -- 172.21.15.53:0/1460033378 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f474400ce70 con 0x7f47540ffcc0 2023-07-14T19:24:29.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f474a7fc700 1 -- 172.21.15.53:0/1460033378 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47440054e0 con 0x7f47540ffcc0 2023-07-14T19:24:29.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.625+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475419db40 con 0x7f47540ffcc0 2023-07-14T19:24:29.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.626+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475419e000 con 0x7f47540ffcc0 2023-07-14T19:24:29.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.626+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475410a2b0 con 0x7f47540ffcc0 2023-07-14T19:24:29.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.628+0000 7f474a7fc700 1 -- 172.21.15.53:0/1460033378 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4744004750 con 0x7f47540ffcc0 2023-07-14T19:24:29.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.628+0000 7f474a7fc700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f47400a11e0 0x7f47400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.628+0000 7f4759597700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f47400a11e0 0x7f47400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.628+0000 7f474a7fc700 1 -- 172.21.15.53:0/1460033378 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f47440cc3f0 con 0x7f47540ffcc0 2023-07-14T19:24:29.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.629+0000 7f4759597700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f47400a11e0 0x7f47400a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f47540ffb20 tx=0x7f475000b670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:29.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.632+0000 7f474a7fc700 1 -- 172.21.15.53:0/1460033378 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4744095b90 con 0x7f47540ffcc0 2023-07-14T19:24:29.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:29 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2732978535' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:24:29.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:29 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2732978535' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:24:29.832 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.831+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4754105930 con 0x7f47540ffcc0 2023-07-14T19:24:29.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.831+0000 7f474a7fc700 1 -- 172.21.15.53:0/1460033378 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f47440991b0 con 0x7f47540ffcc0 2023-07-14T19:24:29.833 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2023-07-14T19:24:29.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f47400a11e0 msgr2=0x7f47400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f47400a11e0 0x7f47400a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f47540ffb20 tx=0x7f475000b670 comp rx=0 tx=0).stop 2023-07-14T19:24:29.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47540ffcc0 msgr2=0x7f475419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47540ffcc0 0x7f475419a750 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f474400c920 tx=0x7f474400cc30 comp rx=0 tx=0).stop 2023-07-14T19:24:29.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 shutdown_connections 2023-07-14T19:24:29.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f47400a11e0 0x7f47400a3690 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4754100650 0x7f475419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47540fead0 0x7f4754068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 --2- 172.21.15.53:0/1460033378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47540ffcc0 0x7f475419a750 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 >> 172.21.15.53:0/1460033378 conn(0x7f47540fa0d0 msgr2=0x7f47540fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:29.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 shutdown_connections 2023-07-14T19:24:29.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.834+0000 7f475b7fb700 1 -- 172.21.15.53:0/1460033378 wait complete. 2023-07-14T19:24:29.847 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:24:29.978 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.974+0000 7fcfd207f700 1 -- 172.21.15.53:0/2972793058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 msgr2=0x7fcfcc100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.978 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.974+0000 7fcfd207f700 1 --2- 172.21.15.53:0/2972793058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 0x7fcfcc100150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009990 tx=0x7fcfc0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.978 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.974+0000 7fcfd207f700 1 -- 172.21.15.53:0/2972793058 shutdown_connections 2023-07-14T19:24:29.978 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.974+0000 7fcfd207f700 1 --2- 172.21.15.53:0/2972793058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc100690 0x7fcfcc106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.978 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.974+0000 7fcfd207f700 1 --2- 172.21.15.53:0/2972793058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfcc0feb10 0x7fcfcc0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.974+0000 7fcfd207f700 1 --2- 172.21.15.53:0/2972793058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 0x7fcfcc100150 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.974+0000 7fcfd207f700 1 -- 172.21.15.53:0/2972793058 >> 172.21.15.53:0/2972793058 conn(0x7fcfcc0fa130 msgr2=0x7fcfcc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:29.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.974+0000 7fcfd207f700 1 -- 172.21.15.53:0/2972793058 shutdown_connections 2023-07-14T19:24:29.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.974+0000 7fcfd207f700 1 -- 172.21.15.53:0/2972793058 wait complete. 2023-07-14T19:24:29.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.975+0000 7fcfd207f700 1 Processor -- start 2023-07-14T19:24:29.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.975+0000 7fcfd207f700 1 -- start start 2023-07-14T19:24:29.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.975+0000 7fcfd207f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0feb10 0x7fcfcc068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.975+0000 7fcfd207f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 0x7fcfcc19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.975+0000 7fcfd207f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfcc100690 0x7fcfcc19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.975+0000 7fcfd207f700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc19d500 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.975+0000 7fcfd207f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc19d670 con 0x7fcfcc0feb10 2023-07-14T19:24:29.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.975+0000 7fcfd207f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc19d830 con 0x7fcfcc100690 2023-07-14T19:24:29.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.975+0000 7fcfcaffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 0x7fcfcc19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcaffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 0x7fcfcc19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50882/0 (socket says 172.21.15.53:50882) 2023-07-14T19:24:29.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcaffd700 1 -- 172.21.15.53:0/782554915 learned_addr learned my addr 172.21.15.53:0/782554915 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:29.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcbfff700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfcc100690 0x7fcfcc19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcb7fe700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0feb10 0x7fcfcc068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcaffd700 1 -- 172.21.15.53:0/782554915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfcc100690 msgr2=0x7fcfcc19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcaffd700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfcc100690 0x7fcfcc19cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcaffd700 1 -- 172.21.15.53:0/782554915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0feb10 msgr2=0x7fcfcc068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:29.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcaffd700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0feb10 0x7fcfcc068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:29.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcaffd700 1 -- 172.21.15.53:0/782554915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00095f0 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcb7fe700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0feb10 0x7fcfcc068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:29.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcbfff700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfcc100690 0x7fcfcc19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:29.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfcaffd700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 0x7fcfcc19a6e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0005910 tx=0x7fcfc0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:29.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfc8ff9700 1 -- 172.21.15.53:0/782554915 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc001d070 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfc8ff9700 1 -- 172.21.15.53:0/782554915 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfc0004060 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfc8ff9700 1 -- 172.21.15.53:0/782554915 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0022e10 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc19dab0 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.976+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc19de80 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.977+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc10a3a0 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.978+0000 7fcfc8ff9700 1 -- 172.21.15.53:0/782554915 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fcfc0022470 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.979+0000 7fcfc8ff9700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcfb40a1260 0x7fcfb40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:29.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.979+0000 7fcfcb7fe700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcfb40a1260 0x7fcfb40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:29.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.979+0000 7fcfc8ff9700 1 -- 172.21.15.53:0/782554915 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcfc00ce280 con 0x7fcfcc0ffd00 2023-07-14T19:24:29.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.979+0000 7fcfcb7fe700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcfb40a1260 0x7fcfb40a3710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc005fd0 tx=0x7fcfbc005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:29.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:29.982+0000 7fcfc8ff9700 1 -- 172.21.15.53:0/782554915 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcfc0097a40 con 0x7fcfcc0ffd00 2023-07-14T19:24:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.181+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcfcc105a10 con 0x7fcfcc0ffd00 2023-07-14T19:24:30.183 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.181+0000 7fcfc8ff9700 1 -- 172.21.15.53:0/782554915 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcfc009b060 con 0x7fcfcc0ffd00 2023-07-14T19:24:30.183 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2023-07-14T19:24:30.185 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.183+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcfb40a1260 msgr2=0x7fcfb40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.185 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.183+0000 7fcfd207f700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcfb40a1260 0x7fcfb40a3710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc005fd0 tx=0x7fcfbc005dc0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.185 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.183+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 msgr2=0x7fcfcc19a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.185 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.183+0000 7fcfd207f700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 0x7fcfcc19a6e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0005910 tx=0x7fcfc0005e80 comp rx=0 tx=0).stop 2023-07-14T19:24:30.185 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.183+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 shutdown_connections 2023-07-14T19:24:30.185 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.183+0000 7fcfd207f700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fcfb40a1260 0x7fcfb40a3710 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.186 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.183+0000 7fcfd207f700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfcc100690 0x7fcfcc19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.186 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.183+0000 7fcfd207f700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0feb10 0x7fcfcc068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.186 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.183+0000 7fcfd207f700 1 --2- 172.21.15.53:0/782554915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcfcc0ffd00 0x7fcfcc19a6e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.186 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.184+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 >> 172.21.15.53:0/782554915 conn(0x7fcfcc0fa130 msgr2=0x7fcfcc0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:30.186 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.184+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 shutdown_connections 2023-07-14T19:24:30.187 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.184+0000 7fcfd207f700 1 -- 172.21.15.53:0/782554915 wait complete. 2023-07-14T19:24:30.197 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:24:30.332 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.325+0000 7fe52c210700 1 -- 172.21.15.53:0/2576686512 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5240ffd30 msgr2=0x7fe524100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.325+0000 7fe52c210700 1 --2- 172.21.15.53:0/2576686512 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5240ffd30 0x7fe524100180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe514009990 tx=0x7fe514009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.325+0000 7fe52c210700 1 -- 172.21.15.53:0/2576686512 shutdown_connections 2023-07-14T19:24:30.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.325+0000 7fe52c210700 1 --2- 172.21.15.53:0/2576686512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5241006c0 0x7fe524106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.334 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.325+0000 7fe52c210700 1 --2- 172.21.15.53:0/2576686512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5240feb40 0x7fe5240fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.334 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.325+0000 7fe52c210700 1 --2- 172.21.15.53:0/2576686512 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5240ffd30 0x7fe524100180 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.334 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.325+0000 7fe52c210700 1 -- 172.21.15.53:0/2576686512 >> 172.21.15.53:0/2576686512 conn(0x7fe5240fa160 msgr2=0x7fe5240fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:30.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 -- 172.21.15.53:0/2576686512 shutdown_connections 2023-07-14T19:24:30.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 -- 172.21.15.53:0/2576686512 wait complete. 2023-07-14T19:24:30.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 Processor -- start 2023-07-14T19:24:30.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 -- start start 2023-07-14T19:24:30.336 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5240feb40 0x7fe524069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:30.336 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5240ffd30 0x7fe52419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:30.336 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5241006c0 0x7fe52419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:30.337 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52419d490 con 0x7fe5241006c0 2023-07-14T19:24:30.337 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52419d600 con 0x7fe5240ffd30 2023-07-14T19:24:30.337 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52c210700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52419d7c0 con 0x7fe5240feb40 2023-07-14T19:24:30.337 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.326+0000 7fe52a7ad700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5241006c0 0x7fe52419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:30.338 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52a7ad700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5241006c0 0x7fe52419ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50898/0 (socket says 172.21.15.53:50898) 2023-07-14T19:24:30.338 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52a7ad700 1 -- 172.21.15.53:0/1731127584 learned_addr learned my addr 172.21.15.53:0/1731127584 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:30.338 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe5297ab700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5240ffd30 0x7fe52419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:30.338 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe529fac700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5240feb40 0x7fe524069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:30.339 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52a7ad700 1 -- 172.21.15.53:0/1731127584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5240feb40 msgr2=0x7fe524069300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.339 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52a7ad700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5240feb40 0x7fe524069300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.339 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52a7ad700 1 -- 172.21.15.53:0/1731127584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5240ffd30 msgr2=0x7fe52419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.340 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52a7ad700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5240ffd30 0x7fe52419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.340 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52a7ad700 1 -- 172.21.15.53:0/1731127584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5140095f0 con 0x7fe5241006c0 2023-07-14T19:24:30.340 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe529fac700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5240feb40 0x7fe524069300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:30.340 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52a7ad700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5241006c0 0x7fe52419ccc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe51c007d70 tx=0x7fe51c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:30.340 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe51affd700 1 -- 172.21.15.53:0/1731127584 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00ad40 con 0x7fe5241006c0 2023-07-14T19:24:30.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52419daa0 con 0x7fe5241006c0 2023-07-14T19:24:30.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe51affd700 1 -- 172.21.15.53:0/1731127584 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe51c0128e0 con 0x7fe5241006c0 2023-07-14T19:24:30.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe51affd700 1 -- 172.21.15.53:0/1731127584 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c011780 con 0x7fe5241006c0 2023-07-14T19:24:30.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.327+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe524194440 con 0x7fe5241006c0 2023-07-14T19:24:30.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.328+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5241059f0 con 0x7fe5241006c0 2023-07-14T19:24:30.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.329+0000 7fe51affd700 1 -- 172.21.15.53:0/1731127584 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe51c012a50 con 0x7fe5241006c0 2023-07-14T19:24:30.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.330+0000 7fe51affd700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe5100a55f0 0x7fe5100a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:30.342 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.330+0000 7fe529fac700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe5100a55f0 0x7fe5100a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:30.342 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.330+0000 7fe51affd700 1 -- 172.21.15.53:0/1731127584 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe51c0ccc10 con 0x7fe5241006c0 2023-07-14T19:24:30.342 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.330+0000 7fe529fac700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe5100a55f0 0x7fe5100a7aa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe5240ffb90 tx=0x7fe520009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:30.342 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.334+0000 7fe51affd700 1 -- 172.21.15.53:0/1731127584 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe51c0963c0 con 0x7fe5241006c0 2023-07-14T19:24:30.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.533+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe52402cb00 con 0x7fe5241006c0 2023-07-14T19:24:30.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.533+0000 7fe51affd700 1 -- 172.21.15.53:0/1731127584 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe51c0160a0 con 0x7fe5241006c0 2023-07-14T19:24:30.535 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2023-07-14T19:24:30.536 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe5100a55f0 msgr2=0x7fe5100a7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe5100a55f0 0x7fe5100a7aa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe5240ffb90 tx=0x7fe520009260 comp rx=0 tx=0).stop 2023-07-14T19:24:30.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5241006c0 msgr2=0x7fe52419ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5241006c0 0x7fe52419ccc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe51c007d70 tx=0x7fe51c00a340 comp rx=0 tx=0).stop 2023-07-14T19:24:30.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 shutdown_connections 2023-07-14T19:24:30.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe5100a55f0 0x7fe5100a7aa0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5241006c0 0x7fe52419ccc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.537 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5240feb40 0x7fe524069300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.538 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 --2- 172.21.15.53:0/1731127584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5240ffd30 0x7fe52419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.538 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 >> 172.21.15.53:0/1731127584 conn(0x7fe5240fa160 msgr2=0x7fe5240fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:30.538 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 shutdown_connections 2023-07-14T19:24:30.538 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.535+0000 7fe52c210700 1 -- 172.21.15.53:0/1731127584 wait complete. 2023-07-14T19:24:30.548 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:30 smithi053 ceph-mon[96565]: pgmap v274: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:24:30.549 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:30 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1468782509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:30.549 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:30 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1460033378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:30.549 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:30 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/782554915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:30.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:24:30.676 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.674+0000 7f1a2705a700 1 -- 172.21.15.53:0/1382676420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a200ffc10 msgr2=0x7f1a20100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.676 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.674+0000 7f1a2705a700 1 --2- 172.21.15.53:0/1382676420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a200ffc10 0x7f1a20100020 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009990 tx=0x7f1a14009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.676 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.674+0000 7f1a2705a700 1 -- 172.21.15.53:0/1382676420 shutdown_connections 2023-07-14T19:24:30.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.674+0000 7f1a2705a700 1 --2- 172.21.15.53:0/1382676420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a20101790 0x7f1a20108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.674+0000 7f1a2705a700 1 --2- 172.21.15.53:0/1382676420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a200ffc10 0x7f1a20100020 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.674+0000 7f1a2705a700 1 --2- 172.21.15.53:0/1382676420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a20100e00 0x7f1a20101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.674+0000 7f1a2705a700 1 -- 172.21.15.53:0/1382676420 >> 172.21.15.53:0/1382676420 conn(0x7f1a200fb230 msgr2=0x7f1a200fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:30.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.674+0000 7f1a2705a700 1 -- 172.21.15.53:0/1382676420 shutdown_connections 2023-07-14T19:24:30.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.674+0000 7f1a2705a700 1 -- 172.21.15.53:0/1382676420 wait complete. 2023-07-14T19:24:30.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.675+0000 7f1a2705a700 1 Processor -- start 2023-07-14T19:24:30.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.675+0000 7f1a2705a700 1 -- start start 2023-07-14T19:24:30.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.675+0000 7f1a2705a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a200ffc10 0x7f1a20195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:30.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.675+0000 7f1a2705a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a20100e00 0x7f1a20195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:30.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.675+0000 7f1a2705a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a20101790 0x7f1a2019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:30.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.675+0000 7f1a2705a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2019d4a0 con 0x7f1a20101790 2023-07-14T19:24:30.681 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.675+0000 7f1a2705a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2019d5e0 con 0x7f1a200ffc10 2023-07-14T19:24:30.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.675+0000 7f1a2705a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2019d770 con 0x7f1a20100e00 2023-07-14T19:24:30.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.675+0000 7f1a255f7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a20101790 0x7f1a2019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:30.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a255f7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a20101790 0x7f1a2019ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50910/0 (socket says 172.21.15.53:50910) 2023-07-14T19:24:30.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a255f7700 1 -- 172.21.15.53:0/4264002660 learned_addr learned my addr 172.21.15.53:0/4264002660 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:30.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a1ffff700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a20100e00 0x7f1a20195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:30.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a24df6700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a200ffc10 0x7f1a20195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:30.682 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a255f7700 1 -- 172.21.15.53:0/4264002660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a20100e00 msgr2=0x7f1a20195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.683 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a255f7700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a20100e00 0x7f1a20195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.683 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a255f7700 1 -- 172.21.15.53:0/4264002660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a200ffc10 msgr2=0x7f1a20195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.683 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a255f7700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a200ffc10 0x7f1a20195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.683 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a255f7700 1 -- 172.21.15.53:0/4264002660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a140095f0 con 0x7f1a20101790 2023-07-14T19:24:30.683 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a24df6700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a200ffc10 0x7f1a20195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:30.683 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a255f7700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a20101790 0x7f1a2019ced0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1a18007ee0 tx=0x7f1a1800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:30.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a1dffb700 1 -- 172.21.15.53:0/4264002660 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18017070 con 0x7f1a20101790 2023-07-14T19:24:30.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a1dffb700 1 -- 172.21.15.53:0/4264002660 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a1800f040 con 0x7f1a20101790 2023-07-14T19:24:30.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2019da50 con 0x7f1a20101790 2023-07-14T19:24:30.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a1dffb700 1 -- 172.21.15.53:0/4264002660 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a18005740 con 0x7f1a20101790 2023-07-14T19:24:30.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.676+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2019df10 con 0x7f1a20101790 2023-07-14T19:24:30.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.677+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a201071b0 con 0x7f1a20101790 2023-07-14T19:24:30.684 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.678+0000 7f1a1dffb700 1 -- 172.21.15.53:0/4264002660 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1a1800dc60 con 0x7f1a20101790 2023-07-14T19:24:30.685 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.679+0000 7f1a1dffb700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1a100a1220 0x7f1a100a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:30.685 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.679+0000 7f1a24df6700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1a100a1220 0x7f1a100a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:30.685 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.679+0000 7f1a1dffb700 1 -- 172.21.15.53:0/4264002660 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1a180cc230 con 0x7f1a20101790 2023-07-14T19:24:30.686 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.680+0000 7f1a24df6700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1a100a1220 0x7f1a100a36d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1a14000c00 tx=0x7f1a14005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:30.686 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.684+0000 7f1a1dffb700 1 -- 172.21.15.53:0/4264002660 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1a180958d0 con 0x7f1a20101790 2023-07-14T19:24:30.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:30 smithi070 ceph-mon[98015]: pgmap v274: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:24:30.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:30 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1468782509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:30.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:30 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1460033378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:30.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:30 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/782554915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:30.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:30 smithi104 ceph-mon[98435]: pgmap v274: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:24:30.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:30 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1468782509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:30.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:30 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1460033378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:30.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:30 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/782554915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:30.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.928+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a2002caf0 con 0x7f1a20101790 2023-07-14T19:24:30.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.928+0000 7f1a1dffb700 1 -- 172.21.15.53:0/4264002660 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f1a18098ef0 con 0x7f1a20101790 2023-07-14T19:24:30.930 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:30.930 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:23:37.894565+0000","modified":"2023-07-14T19:23:43.289341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24491,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24484":{"gid":24484,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24491":{"gid":24491,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6839/2496341573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":2496341573},{"type":"v1","addr":"172.21.15.70:6839","nonce":2496341573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:24:30.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.930+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1a100a1220 msgr2=0x7f1a100a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.933 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.930+0000 7f1a2705a700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1a100a1220 0x7f1a100a36d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1a14000c00 tx=0x7f1a14005fb0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.933 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.930+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a20101790 msgr2=0x7f1a2019ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:30.933 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.930+0000 7f1a2705a700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a20101790 0x7f1a2019ced0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1a18007ee0 tx=0x7f1a1800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.933 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.931+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 shutdown_connections 2023-07-14T19:24:30.934 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.931+0000 7f1a2705a700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1a100a1220 0x7f1a100a36d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.934 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.931+0000 7f1a2705a700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a20101790 0x7f1a2019ced0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.934 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.931+0000 7f1a2705a700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a200ffc10 0x7f1a20195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.934 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.931+0000 7f1a2705a700 1 --2- 172.21.15.53:0/4264002660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a20100e00 0x7f1a20195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:30.934 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.931+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 >> 172.21.15.53:0/4264002660 conn(0x7f1a200fb230 msgr2=0x7f1a201042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:30.934 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.931+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 shutdown_connections 2023-07-14T19:24:30.934 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:30.931+0000 7f1a2705a700 1 -- 172.21.15.53:0/4264002660 wait complete. 2023-07-14T19:24:30.935 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 23 2023-07-14T19:24:30.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:24:31.073 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.068+0000 7f04a4392700 1 -- 172.21.15.53:0/2505244149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c105920 msgr2=0x7f049c107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.068+0000 7f04a4392700 1 --2- 172.21.15.53:0/2505244149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c105920 0x7f049c107d20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f049400b0a0 tx=0x7f049400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 -- 172.21.15.53:0/2505244149 shutdown_connections 2023-07-14T19:24:31.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 --2- 172.21.15.53:0/2505244149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c105920 0x7f049c107d20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 --2- 172.21.15.53:0/2505244149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c1006e0 0x7f049c102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 --2- 172.21.15.53:0/2505244149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c103000 0x7f049c1053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 -- 172.21.15.53:0/2505244149 >> 172.21.15.53:0/2505244149 conn(0x7f049c0fa100 msgr2=0x7f049c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:31.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 -- 172.21.15.53:0/2505244149 shutdown_connections 2023-07-14T19:24:31.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 -- 172.21.15.53:0/2505244149 wait complete. 2023-07-14T19:24:31.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 Processor -- start 2023-07-14T19:24:31.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 -- start start 2023-07-14T19:24:31.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c1006e0 0x7f049c195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c103000 0x7f049c195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c105920 0x7f049c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c19d5f0 con 0x7f049c103000 2023-07-14T19:24:31.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c19d730 con 0x7f049c105920 2023-07-14T19:24:31.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.069+0000 7f04a4392700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c19d8c0 con 0x7f049c1006e0 2023-07-14T19:24:31.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a192d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c103000 0x7f049c195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a192d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c103000 0x7f049c195fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50922/0 (socket says 172.21.15.53:50922) 2023-07-14T19:24:31.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a192d700 1 -- 172.21.15.53:0/3921978608 learned_addr learned my addr 172.21.15.53:0/3921978608 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:31.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a212e700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c1006e0 0x7f049c195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a292f700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c105920 0x7f049c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a192d700 1 -- 172.21.15.53:0/3921978608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c1006e0 msgr2=0x7f049c195a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a192d700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c1006e0 0x7f049c195a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a192d700 1 -- 172.21.15.53:0/3921978608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c105920 msgr2=0x7f049c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a192d700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c105920 0x7f049c19d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a192d700 1 -- 172.21.15.53:0/3921978608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0494009d00 con 0x7f049c103000 2023-07-14T19:24:31.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a192d700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c103000 0x7f049c195fb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f048c00d950 tx=0x7f048c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:31.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04937fe700 1 -- 172.21.15.53:0/3921978608 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c009b40 con 0x7f049c103000 2023-07-14T19:24:31.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04937fe700 1 -- 172.21.15.53:0/3921978608 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f048c009ca0 con 0x7f049c103000 2023-07-14T19:24:31.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04937fe700 1 -- 172.21.15.53:0/3921978608 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c00f620 con 0x7f049c103000 2023-07-14T19:24:31.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c19dba0 con 0x7f049c103000 2023-07-14T19:24:31.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.070+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c19e0b0 con 0x7f049c103000 2023-07-14T19:24:31.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.071+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c0fbc70 con 0x7f049c103000 2023-07-14T19:24:31.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.072+0000 7f04937fe700 1 -- 172.21.15.53:0/3921978608 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f048c010460 con 0x7f049c103000 2023-07-14T19:24:31.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.073+0000 7f04937fe700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f04880a12a0 0x7f04880a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.073+0000 7f04937fe700 1 -- 172.21.15.53:0/3921978608 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f048c0ccc60 con 0x7f049c103000 2023-07-14T19:24:31.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.073+0000 7f04a212e700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f04880a12a0 0x7f04880a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.073+0000 7f04a212e700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f04880a12a0 0x7f04880a3750 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0498009bf0 tx=0x7f0498009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:31.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.076+0000 7f04937fe700 1 -- 172.21.15.53:0/3921978608 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f048c096300 con 0x7f049c103000 2023-07-14T19:24:31.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.275+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f049c02c8e0 con 0x7f049c103000 2023-07-14T19:24:31.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.276+0000 7f04937fe700 1 -- 172.21.15.53:0/3921978608 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22054 (secure 0 0 0) 0x7f048c016090 con 0x7f049c103000 2023-07-14T19:24:31.278 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:31.278 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":80,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:23:39.280266+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:24:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f04880a12a0 msgr2=0x7f04880a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f04880a12a0 0x7f04880a3750 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0498009bf0 tx=0x7f0498009260 comp rx=0 tx=0).stop 2023-07-14T19:24:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c103000 msgr2=0x7f049c195fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c103000 0x7f049c195fb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f048c00d950 tx=0x7f048c00dd10 comp rx=0 tx=0).stop 2023-07-14T19:24:31.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 shutdown_connections 2023-07-14T19:24:31.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f04880a12a0 0x7f04880a3750 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f049c105920 0x7f049c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f049c1006e0 0x7f049c195a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 --2- 172.21.15.53:0/3921978608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f049c103000 0x7f049c195fb0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 >> 172.21.15.53:0/3921978608 conn(0x7f049c0fa100 msgr2=0x7f049c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:31.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 shutdown_connections 2023-07-14T19:24:31.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.279+0000 7f04a4392700 1 -- 172.21.15.53:0/3921978608 wait complete. 2023-07-14T19:24:31.293 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:19:51.775369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:23:35.935538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:23:37.252372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:24:31.294 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:24:31.423 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.421+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/470959856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f18102fb0 msgr2=0x7f7f18105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.423 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.421+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/470959856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f18102fb0 0x7f7f18105390 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009990 tx=0x7f7f08009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.423 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.421+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/470959856 shutdown_connections 2023-07-14T19:24:31.423 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.421+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/470959856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f181058d0 0x7f7f18107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.421+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/470959856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f18100690 0x7f7f18102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.421+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/470959856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f18102fb0 0x7f7f18105390 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.421+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/470959856 >> 172.21.15.53:0/470959856 conn(0x7f7f180fa130 msgr2=0x7f7f180fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:31.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.422+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/470959856 shutdown_connections 2023-07-14T19:24:31.424 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.422+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/470959856 wait complete. 2023-07-14T19:24:31.426 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.422+0000 7f7f1f2ef700 1 Processor -- start 2023-07-14T19:24:31.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.422+0000 7f7f1f2ef700 1 -- start start 2023-07-14T19:24:31.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1f2ef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f18100690 0x7f7f18195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1f2ef700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f18102fb0 0x7f7f18195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1f2ef700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f181058d0 0x7f7f1819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1f2ef700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1819d490 con 0x7f7f181058d0 2023-07-14T19:24:31.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1f2ef700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1819d5d0 con 0x7f7f18102fb0 2023-07-14T19:24:31.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1f2ef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1819d760 con 0x7f7f18100690 2023-07-14T19:24:31.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d88c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f181058d0 0x7f7f1819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d88c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f181058d0 0x7f7f1819cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50936/0 (socket says 172.21.15.53:50936) 2023-07-14T19:24:31.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d88c700 1 -- 172.21.15.53:0/381722161 learned_addr learned my addr 172.21.15.53:0/381722161 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:31.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1c88a700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f18102fb0 0x7f7f18195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d08b700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f18100690 0x7f7f18195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d88c700 1 -- 172.21.15.53:0/381722161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f18100690 msgr2=0x7f7f18195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d88c700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f18100690 0x7f7f18195910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d88c700 1 -- 172.21.15.53:0/381722161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f18102fb0 msgr2=0x7f7f18195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d88c700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f18102fb0 0x7f7f18195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d88c700 1 -- 172.21.15.53:0/381722161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080095f0 con 0x7f7f181058d0 2023-07-14T19:24:31.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d08b700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f18100690 0x7f7f18195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:31.430 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f1d88c700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f181058d0 0x7f7f1819cec0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7f10007ee0 tx=0x7f7f1000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:31.430 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f0e7fc700 1 -- 172.21.15.53:0/381722161 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f10017070 con 0x7f7f181058d0 2023-07-14T19:24:31.430 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.423+0000 7f7f0e7fc700 1 -- 172.21.15.53:0/381722161 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f1000f040 con 0x7f7f181058d0 2023-07-14T19:24:31.430 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.424+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1819da40 con 0x7f7f181058d0 2023-07-14T19:24:31.430 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.424+0000 7f7f0e7fc700 1 -- 172.21.15.53:0/381722161 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f10005740 con 0x7f7f181058d0 2023-07-14T19:24:31.430 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.424+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1819df50 con 0x7f7f181058d0 2023-07-14T19:24:31.430 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.424+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1810b3d0 con 0x7f7f181058d0 2023-07-14T19:24:31.431 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.426+0000 7f7f0e7fc700 1 -- 172.21.15.53:0/381722161 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7f1000dc60 con 0x7f7f181058d0 2023-07-14T19:24:31.431 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.427+0000 7f7f0e7fc700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7f040a1260 0x7f7f040a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.432 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.427+0000 7f7f1d08b700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7f040a1260 0x7f7f040a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.432 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.427+0000 7f7f0e7fc700 1 -- 172.21.15.53:0/381722161 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7f100cc720 con 0x7f7f181058d0 2023-07-14T19:24:31.432 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.427+0000 7f7f1d08b700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7f040a1260 0x7f7f040a3710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7f14005950 tx=0x7f7f140058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:31.432 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.430+0000 7f7f0e7fc700 1 -- 172.21.15.53:0/381722161 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7f10095ec0 con 0x7f7f181058d0 2023-07-14T19:24:31.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:31 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1731127584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:31 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/4264002660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:31.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:31 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3921978608' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:31.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.629+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7f1819e8e0 con 0x7f7f181058d0 2023-07-14T19:24:31.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.629+0000 7f7f0e7fc700 1 -- 172.21.15.53:0/381722161 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f100994e0 con 0x7f7f181058d0 2023-07-14T19:24:31.631 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2023-07-14T19:24:31.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7f040a1260 msgr2=0x7f7f040a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7f040a1260 0x7f7f040a3710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7f14005950 tx=0x7f7f140058e0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f181058d0 msgr2=0x7f7f1819cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f181058d0 0x7f7f1819cec0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7f10007ee0 tx=0x7f7f1000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 shutdown_connections 2023-07-14T19:24:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7f040a1260 0x7f7f040a3710 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f181058d0 0x7f7f1819cec0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7f18100690 0x7f7f18195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 --2- 172.21.15.53:0/381722161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f18102fb0 0x7f7f18195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 >> 172.21.15.53:0/381722161 conn(0x7f7f180fa130 msgr2=0x7f7f18101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 shutdown_connections 2023-07-14T19:24:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.631+0000 7f7f1f2ef700 1 -- 172.21.15.53:0/381722161 wait complete. 2023-07-14T19:24:31.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:24:31.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:31 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1731127584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:31.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:31 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/4264002660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:31.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:31 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3921978608' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:31.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:31 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1731127584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:31.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:31 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/4264002660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:31.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:31 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3921978608' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:31.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 -- 172.21.15.53:0/2824253790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9928100e30 msgr2=0x7f9928101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 --2- 172.21.15.53:0/2824253790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9928100e30 0x7f9928101280 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9918009990 tx=0x7f9918009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 -- 172.21.15.53:0/2824253790 shutdown_connections 2023-07-14T19:24:31.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 --2- 172.21.15.53:0/2824253790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99281017c0 0x7f9928108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 --2- 172.21.15.53:0/2824253790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f99280ffc40 0x7f9928100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 --2- 172.21.15.53:0/2824253790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9928100e30 0x7f9928101280 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 -- 172.21.15.53:0/2824253790 >> 172.21.15.53:0/2824253790 conn(0x7f99280fb260 msgr2=0x7f99280fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:31.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 -- 172.21.15.53:0/2824253790 shutdown_connections 2023-07-14T19:24:31.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 -- 172.21.15.53:0/2824253790 wait complete. 2023-07-14T19:24:31.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.773+0000 7f992de32700 1 Processor -- start 2023-07-14T19:24:31.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f992de32700 1 -- start start 2023-07-14T19:24:31.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f992de32700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280ffc40 0x7f99281959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f992de32700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9928100e30 0x7f9928195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f992de32700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99281017c0 0x7f992819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f992de32700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992819d560 con 0x7f99280ffc40 2023-07-14T19:24:31.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f992de32700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992819d6a0 con 0x7f99281017c0 2023-07-14T19:24:31.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f992de32700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992819d830 con 0x7f9928100e30 2023-07-14T19:24:31.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f99277fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280ffc40 0x7f99281959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f99277fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280ffc40 0x7f99281959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50946/0 (socket says 172.21.15.53:50946) 2023-07-14T19:24:31.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f99277fe700 1 -- 172.21.15.53:0/901646311 learned_addr learned my addr 172.21.15.53:0/901646311 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:31.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f9926ffd700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9928100e30 0x7f9928195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f9927fff700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99281017c0 0x7f992819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f99277fe700 1 -- 172.21.15.53:0/901646311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9928100e30 msgr2=0x7f9928195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f99277fe700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9928100e30 0x7f9928195f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f99277fe700 1 -- 172.21.15.53:0/901646311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99281017c0 msgr2=0x7f992819cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f99277fe700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99281017c0 0x7f992819cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f99277fe700 1 -- 172.21.15.53:0/901646311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99180095f0 con 0x7f99280ffc40 2023-07-14T19:24:31.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f99277fe700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280ffc40 0x7f99281959e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f991c00c8f0 tx=0x7f991c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:31.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f9924ff9700 1 -- 172.21.15.53:0/901646311 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c004220 con 0x7f99280ffc40 2023-07-14T19:24:31.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.774+0000 7f9924ff9700 1 -- 172.21.15.53:0/901646311 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f991c004380 con 0x7f99280ffc40 2023-07-14T19:24:31.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.775+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992819db10 con 0x7f99280ffc40 2023-07-14T19:24:31.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.775+0000 7f9924ff9700 1 -- 172.21.15.53:0/901646311 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c003b40 con 0x7f99280ffc40 2023-07-14T19:24:31.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.775+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992819e020 con 0x7f99280ffc40 2023-07-14T19:24:31.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.775+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99281071d0 con 0x7f99280ffc40 2023-07-14T19:24:31.781 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.777+0000 7f9924ff9700 1 -- 172.21.15.53:0/901646311 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f991c003d00 con 0x7f99280ffc40 2023-07-14T19:24:31.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.777+0000 7f9924ff9700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99100a1260 0x7f99100a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:31.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.777+0000 7f9924ff9700 1 -- 172.21.15.53:0/901646311 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f991c0cc7a0 con 0x7f99280ffc40 2023-07-14T19:24:31.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.777+0000 7f9926ffd700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99100a1260 0x7f99100a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:31.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.778+0000 7f9926ffd700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99100a1260 0x7f99100a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f991800b5c0 tx=0x7f9918005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:31.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.780+0000 7f9924ff9700 1 -- 172.21.15.53:0/901646311 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f991c095fc0 con 0x7f99280ffc40 2023-07-14T19:24:31.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.980+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f992802ca70 con 0x7f99280ffc40 2023-07-14T19:24:31.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.980+0000 7f9924ff9700 1 -- 172.21.15.53:0/901646311 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v80) v1 ==== 104+0+11 (secure 0 0 0) 0x7f991c01d0a0 con 0x7f99280ffc40 2023-07-14T19:24:31.982 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2023-07-14T19:24:31.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99100a1260 msgr2=0x7f99100a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99100a1260 0x7f99100a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f991800b5c0 tx=0x7f9918005fb0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280ffc40 msgr2=0x7f99281959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:31.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280ffc40 0x7f99281959e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f991c00c8f0 tx=0x7f991c00cc00 comp rx=0 tx=0).stop 2023-07-14T19:24:31.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 shutdown_connections 2023-07-14T19:24:31.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f99100a1260 0x7f99100a3710 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99281017c0 0x7f992819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99280ffc40 0x7f99281959e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 --2- 172.21.15.53:0/901646311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9928100e30 0x7f9928195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:31.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.982+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 >> 172.21.15.53:0/901646311 conn(0x7f99280fb260 msgr2=0x7f9928104310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:31.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.983+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 shutdown_connections 2023-07-14T19:24:31.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:31.983+0000 7f992de32700 1 -- 172.21.15.53:0/901646311 wait complete. 2023-07-14T19:24:31.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:24:32.125 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.123+0000 7f1fd34c8700 1 -- 172.21.15.53:0/2759315505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0ffd00 msgr2=0x7f1fcc100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.125 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.123+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/2759315505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0ffd00 0x7f1fcc100150 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009990 tx=0x7f1fbc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.125 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.123+0000 7f1fd34c8700 1 -- 172.21.15.53:0/2759315505 shutdown_connections 2023-07-14T19:24:32.125 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.123+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/2759315505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fcc100690 0x7f1fcc106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.123+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/2759315505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fcc0feb10 0x7f1fcc0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.123+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/2759315505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0ffd00 0x7f1fcc100150 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.123+0000 7f1fd34c8700 1 -- 172.21.15.53:0/2759315505 >> 172.21.15.53:0/2759315505 conn(0x7f1fcc0fa130 msgr2=0x7f1fcc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.123+0000 7f1fd34c8700 1 -- 172.21.15.53:0/2759315505 shutdown_connections 2023-07-14T19:24:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.123+0000 7f1fd34c8700 1 -- 172.21.15.53:0/2759315505 wait complete. 2023-07-14T19:24:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd34c8700 1 Processor -- start 2023-07-14T19:24:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd34c8700 1 -- start start 2023-07-14T19:24:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd34c8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0feb10 0x7f1fcc068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd34c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fcc0ffd00 0x7f1fcc19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd34c8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fcc100690 0x7f1fcc19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd34c8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc19d500 con 0x7f1fcc0feb10 2023-07-14T19:24:32.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd34c8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc19d670 con 0x7f1fcc0ffd00 2023-07-14T19:24:32.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd34c8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc19d830 con 0x7f1fcc100690 2023-07-14T19:24:32.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd1264700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0feb10 0x7f1fcc068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd1264700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0feb10 0x7f1fcc068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50960/0 (socket says 172.21.15.53:50960) 2023-07-14T19:24:32.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd1264700 1 -- 172.21.15.53:0/1368425110 learned_addr learned my addr 172.21.15.53:0/1368425110 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:32.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd0a63700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fcc0ffd00 0x7f1fcc19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd1264700 1 -- 172.21.15.53:0/1368425110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fcc100690 msgr2=0x7f1fcc19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd1264700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fcc100690 0x7f1fcc19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd1264700 1 -- 172.21.15.53:0/1368425110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fcc0ffd00 msgr2=0x7f1fcc19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd1264700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fcc0ffd00 0x7f1fcc19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd1264700 1 -- 172.21.15.53:0/1368425110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc0095f0 con 0x7f1fcc0feb10 2023-07-14T19:24:32.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.124+0000 7f1fd0a63700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fcc0ffd00 0x7f1fcc19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:32.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.125+0000 7f1fd1264700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0feb10 0x7f1fcc068330 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800d980 tx=0x7f1fc800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:32.131 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.125+0000 7f1fc27fc700 1 -- 172.21.15.53:0/1368425110 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc800de60 con 0x7f1fcc0feb10 2023-07-14T19:24:32.131 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.125+0000 7f1fc27fc700 1 -- 172.21.15.53:0/1368425110 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fc80044e0 con 0x7f1fcc0feb10 2023-07-14T19:24:32.131 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.125+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc19db10 con 0x7f1fcc0feb10 2023-07-14T19:24:32.131 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.125+0000 7f1fc27fc700 1 -- 172.21.15.53:0/1368425110 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8003be0 con 0x7f1fcc0feb10 2023-07-14T19:24:32.131 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.125+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc19e020 con 0x7f1fcc0feb10 2023-07-14T19:24:32.131 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.126+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc10a330 con 0x7f1fcc0feb10 2023-07-14T19:24:32.131 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.127+0000 7f1fc27fc700 1 -- 172.21.15.53:0/1368425110 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1fc8009bb0 con 0x7f1fcc0feb10 2023-07-14T19:24:32.132 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.127+0000 7f1fc27fc700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1fb80a11e0 0x7f1fb80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.132 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.127+0000 7f1fd0a63700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1fb80a11e0 0x7f1fb80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.127+0000 7f1fc27fc700 1 -- 172.21.15.53:0/1368425110 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1fc80cca30 con 0x7f1fcc0feb10 2023-07-14T19:24:32.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.128+0000 7f1fd0a63700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1fb80a11e0 0x7f1fb80a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc005f50 tx=0x7f1fbc005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:32.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.131+0000 7f1fc27fc700 1 -- 172.21.15.53:0/1368425110 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1fc80961d0 con 0x7f1fcc0feb10 2023-07-14T19:24:32.330 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.329+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1fcc105a10 con 0x7f1fcc0feb10 2023-07-14T19:24:32.331 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.329+0000 7f1fc27fc700 1 -- 172.21.15.53:0/1368425110 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v80) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1fc8009940 con 0x7f1fcc0feb10 2023-07-14T19:24:32.331 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2023-07-14T19:24:32.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1fb80a11e0 msgr2=0x7f1fb80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1fb80a11e0 0x7f1fb80a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc005f50 tx=0x7f1fbc005e80 comp rx=0 tx=0).stop 2023-07-14T19:24:32.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0feb10 msgr2=0x7f1fcc068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.333 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0feb10 0x7f1fcc068330 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800d980 tx=0x7f1fc800dd40 comp rx=0 tx=0).stop 2023-07-14T19:24:32.334 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 shutdown_connections 2023-07-14T19:24:32.334 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f1fb80a11e0 0x7f1fb80a3690 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.334 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1fcc100690 0x7f1fcc19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.334 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fcc0feb10 0x7f1fcc068330 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 --2- 172.21.15.53:0/1368425110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fcc0ffd00 0x7f1fcc19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 >> 172.21.15.53:0/1368425110 conn(0x7f1fcc0fa130 msgr2=0x7f1fcc0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:32.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 shutdown_connections 2023-07-14T19:24:32.335 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.332+0000 7f1fd34c8700 1 -- 172.21.15.53:0/1368425110 wait complete. 2023-07-14T19:24:32.346 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:24:32.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.471+0000 7f31bff2c700 1 -- 172.21.15.53:0/1319721774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 msgr2=0x7f31b8106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.471+0000 7f31bff2c700 1 --2- 172.21.15.53:0/1319721774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 0x7f31b8106f40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f31b000b0a0 tx=0x7f31b000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.471+0000 7f31bff2c700 1 -- 172.21.15.53:0/1319721774 shutdown_connections 2023-07-14T19:24:32.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.471+0000 7f31bff2c700 1 --2- 172.21.15.53:0/1319721774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 0x7f31b8106f40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.471+0000 7f31bff2c700 1 --2- 172.21.15.53:0/1319721774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31b80feb00 0x7f31b80fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.471+0000 7f31bff2c700 1 --2- 172.21.15.53:0/1319721774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31b80ffcf0 0x7f31b8100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.471+0000 7f31bff2c700 1 -- 172.21.15.53:0/1319721774 >> 172.21.15.53:0/1319721774 conn(0x7f31b80fa100 msgr2=0x7f31b80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:32.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.471+0000 7f31bff2c700 1 -- 172.21.15.53:0/1319721774 shutdown_connections 2023-07-14T19:24:32.473 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.471+0000 7f31bff2c700 1 -- 172.21.15.53:0/1319721774 wait complete. 2023-07-14T19:24:32.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bff2c700 1 Processor -- start 2023-07-14T19:24:32.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bff2c700 1 -- start start 2023-07-14T19:24:32.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bff2c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31b80feb00 0x7f31b81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bff2c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31b80ffcf0 0x7f31b8107a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bff2c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 0x7f31b8107f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bff2c700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b8108710 con 0x7f31b8100680 2023-07-14T19:24:32.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bff2c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b8108880 con 0x7f31b80feb00 2023-07-14T19:24:32.476 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bff2c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b819fa80 con 0x7f31b80ffcf0 2023-07-14T19:24:32.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31be4c9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 0x7f31b8107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31be4c9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 0x7f31b8107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50978/0 (socket says 172.21.15.53:50978) 2023-07-14T19:24:32.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31be4c9700 1 -- 172.21.15.53:0/4261154874 learned_addr learned my addr 172.21.15.53:0/4261154874 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:32.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bd4c7700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31b80ffcf0 0x7f31b8107a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31bdcc8700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31b80feb00 0x7f31b81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31be4c9700 1 -- 172.21.15.53:0/4261154874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31b80ffcf0 msgr2=0x7f31b8107a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31be4c9700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31b80ffcf0 0x7f31b8107a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31be4c9700 1 -- 172.21.15.53:0/4261154874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31b80feb00 msgr2=0x7f31b81093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31be4c9700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31b80feb00 0x7f31b81093f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31be4c9700 1 -- 172.21.15.53:0/4261154874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0009d00 con 0x7f31b8100680 2023-07-14T19:24:32.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.472+0000 7f31be4c9700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 0x7f31b8107f70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f31b000d010 tx=0x7f31b0004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:32.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.473+0000 7f31aeffd700 1 -- 172.21.15.53:0/4261154874 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b000e040 con 0x7f31b8100680 2023-07-14T19:24:32.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.473+0000 7f31aeffd700 1 -- 172.21.15.53:0/4261154874 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31b0009440 con 0x7f31b8100680 2023-07-14T19:24:32.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.473+0000 7f31aeffd700 1 -- 172.21.15.53:0/4261154874 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b0004020 con 0x7f31b8100680 2023-07-14T19:24:32.479 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.473+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b819fd00 con 0x7f31b8100680 2023-07-14T19:24:32.479 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.473+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b81a0130 con 0x7f31b8100680 2023-07-14T19:24:32.479 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.474+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b8105940 con 0x7f31b8100680 2023-07-14T19:24:32.479 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.475+0000 7f31aeffd700 1 -- 172.21.15.53:0/4261154874 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f31b001d690 con 0x7f31b8100680 2023-07-14T19:24:32.480 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.476+0000 7f31aeffd700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31a40a12a0 0x7f31a40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.480 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.476+0000 7f31bdcc8700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31a40a12a0 0x7f31a40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.480 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.476+0000 7f31aeffd700 1 -- 172.21.15.53:0/4261154874 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f31b00cebe0 con 0x7f31b8100680 2023-07-14T19:24:32.480 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.476+0000 7f31bdcc8700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31a40a12a0 0x7f31a40a3750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f31b4005950 tx=0x7f31b40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:32.481 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.479+0000 7f31aeffd700 1 -- 172.21.15.53:0/4261154874 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f31b0098390 con 0x7f31b8100680 2023-07-14T19:24:32.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:32 smithi053 ceph-mon[96565]: pgmap v275: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T19:24:32.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:32 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/381722161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:32.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:32 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/901646311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:32.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:32 smithi070 ceph-mon[98015]: pgmap v275: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T19:24:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:32 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/381722161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:32.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:32 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/901646311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:32.729 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.726+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31b802c8e0 con 0x7f31b8100680 2023-07-14T19:24:32.729 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.727+0000 7f31aeffd700 1 -- 172.21.15.53:0/4261154874 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f31b0021090 con 0x7f31b8100680 2023-07-14T19:24:32.729 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:32.729 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:23:37.894565+0000","modified":"2023-07-14T19:23:43.289341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24491,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24484":{"gid":24484,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24491":{"gid":24491,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6839/2496341573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":2496341573},{"type":"v1","addr":"172.21.15.70:6839","nonce":2496341573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:24:32.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31a40a12a0 msgr2=0x7f31a40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31a40a12a0 0x7f31a40a3750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f31b4005950 tx=0x7f31b40058e0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 msgr2=0x7f31b8107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 0x7f31b8107f70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f31b000d010 tx=0x7f31b0004c80 comp rx=0 tx=0).stop 2023-07-14T19:24:32.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 shutdown_connections 2023-07-14T19:24:32.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31a40a12a0 0x7f31a40a3750 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.731 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31b8100680 0x7f31b8107f70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.732 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31b80feb00 0x7f31b81093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.732 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 --2- 172.21.15.53:0/4261154874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31b80ffcf0 0x7f31b8107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.732 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 >> 172.21.15.53:0/4261154874 conn(0x7f31b80fa100 msgr2=0x7f31b80fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:32.732 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 shutdown_connections 2023-07-14T19:24:32.732 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.729+0000 7f31bff2c700 1 -- 172.21.15.53:0/4261154874 wait complete. 2023-07-14T19:24:32.732 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 23 2023-07-14T19:24:32.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:24:32.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:32 smithi104 ceph-mon[98435]: pgmap v275: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T19:24:32.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:32 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/381722161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:32.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:32 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/901646311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.867+0000 7f8a107c8700 1 -- 172.21.15.53:0/3851977870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a08100e20 msgr2=0x7f8a08101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.867+0000 7f8a107c8700 1 --2- 172.21.15.53:0/3851977870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a08100e20 0x7f8a08101270 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009990 tx=0x7f89f8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.867+0000 7f8a107c8700 1 -- 172.21.15.53:0/3851977870 shutdown_connections 2023-07-14T19:24:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.867+0000 7f8a107c8700 1 --2- 172.21.15.53:0/3851977870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a081017b0 0x7f8a08108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.867+0000 7f8a107c8700 1 --2- 172.21.15.53:0/3851977870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a080ffc30 0x7f8a08100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.867+0000 7f8a107c8700 1 --2- 172.21.15.53:0/3851977870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a08100e20 0x7f8a08101270 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.867+0000 7f8a107c8700 1 -- 172.21.15.53:0/3851977870 >> 172.21.15.53:0/3851977870 conn(0x7f8a080fb230 msgr2=0x7f8a080fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:32.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 -- 172.21.15.53:0/3851977870 shutdown_connections 2023-07-14T19:24:32.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 -- 172.21.15.53:0/3851977870 wait complete. 2023-07-14T19:24:32.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 Processor -- start 2023-07-14T19:24:32.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 -- start start 2023-07-14T19:24:32.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a080ffc30 0x7f8a080743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a08100e20 0x7f8a08072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a081017b0 0x7f8a08072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a08073710 con 0x7f8a081017b0 2023-07-14T19:24:32.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a08073880 con 0x7f8a080ffc30 2023-07-14T19:24:32.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.868+0000 7f8a107c8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0819f920 con 0x7f8a08100e20 2023-07-14T19:24:32.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0ed65700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a081017b0 0x7f8a08072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0ed65700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a081017b0 0x7f8a08072f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50992/0 (socket says 172.21.15.53:50992) 2023-07-14T19:24:32.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0ed65700 1 -- 172.21.15.53:0/558883822 learned_addr learned my addr 172.21.15.53:0/558883822 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:32.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0dd63700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a08100e20 0x7f8a08072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0e564700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a080ffc30 0x7f8a080743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0ed65700 1 -- 172.21.15.53:0/558883822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a08100e20 msgr2=0x7f8a08072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0ed65700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a08100e20 0x7f8a08072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0ed65700 1 -- 172.21.15.53:0/558883822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a080ffc30 msgr2=0x7f8a080743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:32.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0ed65700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a080ffc30 0x7f8a080743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:32.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0ed65700 1 -- 172.21.15.53:0/558883822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f80095f0 con 0x7f8a081017b0 2023-07-14T19:24:32.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0e564700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a080ffc30 0x7f8a080743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:32.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a0ed65700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a081017b0 0x7f8a08072f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8a00007ee0 tx=0x7f8a0000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:32.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f89ff7fe700 1 -- 172.21.15.53:0/558883822 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a00017070 con 0x7f8a081017b0 2023-07-14T19:24:32.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f89ff7fe700 1 -- 172.21.15.53:0/558883822 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a0000f040 con 0x7f8a081017b0 2023-07-14T19:24:32.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0819fc00 con 0x7f8a081017b0 2023-07-14T19:24:32.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f89ff7fe700 1 -- 172.21.15.53:0/558883822 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a00005740 con 0x7f8a081017b0 2023-07-14T19:24:32.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.869+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a081a00c0 con 0x7f8a081017b0 2023-07-14T19:24:32.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.870+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a08107150 con 0x7f8a081017b0 2023-07-14T19:24:32.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.871+0000 7f89ff7fe700 1 -- 172.21.15.53:0/558883822 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8a0000dc60 con 0x7f8a081017b0 2023-07-14T19:24:32.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.872+0000 7f89ff7fe700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89f40a1220 0x7f89f40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:32.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.872+0000 7f8a0e564700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89f40a1220 0x7f89f40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:32.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.872+0000 7f89ff7fe700 1 -- 172.21.15.53:0/558883822 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8a000cc740 con 0x7f8a081017b0 2023-07-14T19:24:32.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.872+0000 7f8a0e564700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89f40a1220 0x7f89f40a36d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8a08100c80 tx=0x7f8a04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:32.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:32.875+0000 7f89ff7fe700 1 -- 172.21.15.53:0/558883822 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a00095ee0 con 0x7f8a081017b0 2023-07-14T19:24:33.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.074+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a0802cb30 con 0x7f8a081017b0 2023-07-14T19:24:33.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.076+0000 7f89ff7fe700 1 -- 172.21.15.53:0/558883822 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+22054 (secure 0 0 0) 0x7f8a00099500 con 0x7f8a081017b0 2023-07-14T19:24:33.080 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:33.080 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":80,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:23:39.280266+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:24:33.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89f40a1220 msgr2=0x7f89f40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89f40a1220 0x7f89f40a36d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8a08100c80 tx=0x7f8a04009260 comp rx=0 tx=0).stop 2023-07-14T19:24:33.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a081017b0 msgr2=0x7f8a08072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a081017b0 0x7f8a08072f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8a00007ee0 tx=0x7f8a0000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 shutdown_connections 2023-07-14T19:24:33.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f89f40a1220 0x7f89f40a36d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a081017b0 0x7f8a08072f70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a080ffc30 0x7f8a080743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 --2- 172.21.15.53:0/558883822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a08100e20 0x7f8a08072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 >> 172.21.15.53:0/558883822 conn(0x7f8a080fb230 msgr2=0x7f8a08104290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 shutdown_connections 2023-07-14T19:24:33.083 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.078+0000 7f8a107c8700 1 -- 172.21.15.53:0/558883822 wait complete. 2023-07-14T19:24:33.092 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:24:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.216+0000 7f54b1543700 1 -- 172.21.15.53:0/2223069246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac100680 msgr2=0x7f54ac106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.216+0000 7f54b1543700 1 --2- 172.21.15.53:0/2223069246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac100680 0x7f54ac106f40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f54a400b0a0 tx=0x7f54a400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.216+0000 7f54b1543700 1 -- 172.21.15.53:0/2223069246 shutdown_connections 2023-07-14T19:24:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.216+0000 7f54b1543700 1 --2- 172.21.15.53:0/2223069246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac100680 0x7f54ac106f40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.216+0000 7f54b1543700 1 --2- 172.21.15.53:0/2223069246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54ac0feb00 0x7f54ac0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.216+0000 7f54b1543700 1 --2- 172.21.15.53:0/2223069246 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54ac0ffcf0 0x7f54ac100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.216+0000 7f54b1543700 1 -- 172.21.15.53:0/2223069246 >> 172.21.15.53:0/2223069246 conn(0x7f54ac0fa100 msgr2=0x7f54ac0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.216+0000 7f54b1543700 1 -- 172.21.15.53:0/2223069246 shutdown_connections 2023-07-14T19:24:33.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.216+0000 7f54b1543700 1 -- 172.21.15.53:0/2223069246 wait complete. 2023-07-14T19:24:33.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54b1543700 1 Processor -- start 2023-07-14T19:24:33.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54b1543700 1 -- start start 2023-07-14T19:24:33.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54b1543700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54ac0feb00 0x7f54ac102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54b1543700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac0ffcf0 0x7f54ac101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54b1543700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54ac100680 0x7f54ac1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54b1543700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac101f50 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54b1543700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac1020c0 con 0x7f54ac100680 2023-07-14T19:24:33.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54b1543700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac19f9c0 con 0x7f54ac0feb00 2023-07-14T19:24:33.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54aa7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac0ffcf0 0x7f54ac101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54aa7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac0ffcf0 0x7f54ac101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51004/0 (socket says 172.21.15.53:51004) 2023-07-14T19:24:33.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54aa7fc700 1 -- 172.21.15.53:0/2855393828 learned_addr learned my addr 172.21.15.53:0/2855393828 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54aa7fc700 1 -- 172.21.15.53:0/2855393828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54ac0feb00 msgr2=0x7f54ac102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54aaffd700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54ac0feb00 0x7f54ac102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.217+0000 7f54ab7fe700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54ac100680 0x7f54ac1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f54aa7fc700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54ac0feb00 0x7f54ac102c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f54aa7fc700 1 -- 172.21.15.53:0/2855393828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54ac100680 msgr2=0x7f54ac1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f54aa7fc700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54ac100680 0x7f54ac1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f54aa7fc700 1 -- 172.21.15.53:0/2855393828 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a4009d00 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f54ab7fe700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54ac100680 0x7f54ac1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f54aa7fc700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac0ffcf0 0x7f54ac101270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f54a000c9b0 tx=0x7f54a000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f5493fff700 1 -- 172.21.15.53:0/2855393828 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0007b00 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f5493fff700 1 -- 172.21.15.53:0/2855393828 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54a0007c60 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ac19fca0 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f5493fff700 1 -- 172.21.15.53:0/2855393828 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0018660 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.218+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ac1a01b0 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.219+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac106a70 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.220+0000 7f5493fff700 1 -- 172.21.15.53:0/2855393828 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f54a0018880 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.221+0000 7f5493fff700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f54940a1220 0x7f54940a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.221+0000 7f54aaffd700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f54940a1220 0x7f54940a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.221+0000 7f5493fff700 1 -- 172.21.15.53:0/2855393828 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f54a00cc9b0 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.221+0000 7f54aaffd700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f54940a1220 0x7f54940a36d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f549c005950 tx=0x7f549c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.225+0000 7f5493fff700 1 -- 172.21.15.53:0/2855393828 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f54a0096150 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.470+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54ac02c8e0 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.470+0000 7f5493fff700 1 -- 172.21.15.53:0/2855393828 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f54a0099770 con 0x7f54ac0ffcf0 2023-07-14T19:24:33.472 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:33.472 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:23:37.894565+0000","modified":"2023-07-14T19:23:43.289341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24491,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24484":{"gid":24484,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24491":{"gid":24491,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6839/2496341573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":2496341573},{"type":"v1","addr":"172.21.15.70:6839","nonce":2496341573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:24:33.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.472+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f54940a1220 msgr2=0x7f54940a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.472+0000 7f54b1543700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f54940a1220 0x7f54940a36d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f549c005950 tx=0x7f549c0058e0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.472+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac0ffcf0 msgr2=0x7f54ac101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.472+0000 7f54b1543700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac0ffcf0 0x7f54ac101270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f54a000c9b0 tx=0x7f54a000cd70 comp rx=0 tx=0).stop 2023-07-14T19:24:33.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.473+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 shutdown_connections 2023-07-14T19:24:33.474 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.473+0000 7f54b1543700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f54940a1220 0x7f54940a36d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.473+0000 7f54b1543700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54ac100680 0x7f54ac1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.473+0000 7f54b1543700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54ac0feb00 0x7f54ac102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.473+0000 7f54b1543700 1 --2- 172.21.15.53:0/2855393828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54ac0ffcf0 0x7f54ac101270 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.473+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 >> 172.21.15.53:0/2855393828 conn(0x7f54ac0fa100 msgr2=0x7f54ac0fc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.473+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 shutdown_connections 2023-07-14T19:24:33.475 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.473+0000 7f54b1543700 1 -- 172.21.15.53:0/2855393828 wait complete. 2023-07-14T19:24:33.476 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 23 2023-07-14T19:24:33.485 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:33 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1368425110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:33.485 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:33 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/4261154874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:33.486 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:33 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/558883822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:33.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T19:23:37.894565+0000', 'modified': '2023-07-14T19:23:43.289341+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24527, 'mds_1': 24491, 'mds_2': 24461}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6835/1013468175', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6834', 'nonce': 1013468175}, {'type': 'v1', 'addr': '172.21.15.104:6835', 'nonce': 1013468175}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24461': {'gid': 24461, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.70:6835/2384359054', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6834', 'nonce': 2384359054}, {'type': 'v1', 'addr': '172.21.15.70:6835', 'nonce': 2384359054}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6837/1255142015', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6836', 'nonce': 1255142015}, {'type': 'v1', 'addr': '172.21.15.104:6837', 'nonce': 1255142015}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6839/1650908884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 1650908884}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 1650908884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24491': {'gid': 24491, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.70:6837/3730963239', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 3730963239}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3730963239}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6839/2496341573', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 2496341573}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 2496341573}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T19:24:33.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-14T19:24:33.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-07-14T19:24:33.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.612+0000 7f48dff63700 1 -- 172.21.15.53:0/3974180675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 msgr2=0x7f48d8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.612+0000 7f48dff63700 1 --2- 172.21.15.53:0/3974180675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 0x7f48d8106f50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f48d000b0a0 tx=0x7f48d000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 -- 172.21.15.53:0/3974180675 shutdown_connections 2023-07-14T19:24:33.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 --2- 172.21.15.53:0/3974180675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 0x7f48d8106f50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 --2- 172.21.15.53:0/3974180675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48d80feb10 0x7f48d80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 --2- 172.21.15.53:0/3974180675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d80ffd00 0x7f48d8100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 -- 172.21.15.53:0/3974180675 >> 172.21.15.53:0/3974180675 conn(0x7f48d80fa130 msgr2=0x7f48d80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 -- 172.21.15.53:0/3974180675 shutdown_connections 2023-07-14T19:24:33.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 -- 172.21.15.53:0/3974180675 wait complete. 2023-07-14T19:24:33.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 Processor -- start 2023-07-14T19:24:33.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 -- start start 2023-07-14T19:24:33.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48d80feb10 0x7f48d8069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d80ffd00 0x7f48d819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 0x7f48d819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d819d5e0 con 0x7f48d8100690 2023-07-14T19:24:33.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d819d750 con 0x7f48d80ffd00 2023-07-14T19:24:33.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.613+0000 7f48dff63700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d819d910 con 0x7f48d80feb10 2023-07-14T19:24:33.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48de500700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 0x7f48d819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48de500700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 0x7f48d819ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51020/0 (socket says 172.21.15.53:51020) 2023-07-14T19:24:33.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48de500700 1 -- 172.21.15.53:0/1725230553 learned_addr learned my addr 172.21.15.53:0/1725230553 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d80ffd00 0x7f48d819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48ddcff700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48d80feb10 0x7f48d8069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48de500700 1 -- 172.21.15.53:0/1725230553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48d80feb10 msgr2=0x7f48d8069360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.620 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48de500700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48d80feb10 0x7f48d8069360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48de500700 1 -- 172.21.15.53:0/1725230553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d80ffd00 msgr2=0x7f48d819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48de500700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d80ffd00 0x7f48d819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48de500700 1 -- 172.21.15.53:0/1725230553 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d0009d00 con 0x7f48d8100690 2023-07-14T19:24:33.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48de500700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 0x7f48d819ce10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f48d000d010 tx=0x7f48d000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48ceffd700 1 -- 172.21.15.53:0/1725230553 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d000e040 con 0x7f48d8100690 2023-07-14T19:24:33.621 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48dff63700 1 -- 172.21.15.53:0/1725230553 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d819db90 con 0x7f48d8100690 2023-07-14T19:24:33.622 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48ceffd700 1 -- 172.21.15.53:0/1725230553 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d0003e10 con 0x7f48d8100690 2023-07-14T19:24:33.622 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48ceffd700 1 -- 172.21.15.53:0/1725230553 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d0005260 con 0x7f48d8100690 2023-07-14T19:24:33.622 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.614+0000 7f48dff63700 1 -- 172.21.15.53:0/1725230553 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d819dfb0 con 0x7f48d8100690 2023-07-14T19:24:33.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.616+0000 7f48ceffd700 1 -- 172.21.15.53:0/1725230553 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f48d0005400 con 0x7f48d8100690 2023-07-14T19:24:33.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.617+0000 7f48ceffd700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48c40a11e0 0x7f48c40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.617+0000 7f48ddcff700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48c40a11e0 0x7f48c40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.617+0000 7f48ceffd700 1 -- 172.21.15.53:0/1725230553 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f48d00cdb00 con 0x7f48d8100690 2023-07-14T19:24:33.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.617+0000 7f48ddcff700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48c40a11e0 0x7f48c40a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f48d80ffb60 tx=0x7f48d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ec0fbb80 1 Processor -- start 2023-07-14T19:24:33.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ec0fbb80 1 -- start start 2023-07-14T19:24:33.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a216e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a222c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a22cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.628 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x555710921c60 con 0x555710a212d0 2023-07-14T19:24:33.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557109749f0 con 0x555710a22850 2023-07-14T19:24:33.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55571096de80 con 0x555710a21c20 2023-07-14T19:24:33.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ddcff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a216e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ddcff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a216e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51024/0 (socket says 172.21.15.53:51024) 2023-07-14T19:24:33.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ddcff700 1 -- 172.21.15.53:0/2124738764 learned_addr learned my addr 172.21.15.53:0/2124738764 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48de500700 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a22cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a222c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ddcff700 1 -- 172.21.15.53:0/2124738764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 msgr2=0x555710a222c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ddcff700 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a222c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ddcff700 1 -- 172.21.15.53:0/2124738764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 msgr2=0x555710a22cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ddcff700 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a22cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48ddcff700 1 -- 172.21.15.53:0/2124738764 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55571079bd90 con 0x555710a212d0 2023-07-14T19:24:33.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.618+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a222c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:33.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48ddcff700 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a216e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f48d4011e20 tx=0x7f48d4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4deb04f7cc3d67 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48c3fff700 1 -- 172.21.15.53:0/2124738764 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d4012dc0 con 0x555710a212d0 2023-07-14T19:24:33.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48c3fff700 1 -- 172.21.15.53:0/2124738764 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d4016070 con 0x555710a212d0 2023-07-14T19:24:33.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48c3fff700 1 -- 172.21.15.53:0/2124738764 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d401d610 con 0x555710a212d0 2023-07-14T19:24:33.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/2124738764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 msgr2=0x555710a216e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a216e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f48d4011e20 tx=0x7f48d4014000 comp rx=0 tx=0).stop 2023-07-14T19:24:33.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/2124738764 shutdown_connections 2023-07-14T19:24:33.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a22cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a216e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/2124738764 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a222c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.619+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/2124738764 >> 172.21.15.53:0/2124738764 conn(0x555710a12630 msgr2=0x555710a17980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/2124738764 shutdown_connections 2023-07-14T19:24:33.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/2124738764 wait complete. 2023-07-14T19:24:33.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 -- start start 2023-07-14T19:24:33.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 Processor -- start 2023-07-14T19:24:33.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a2ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a2b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a32280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55571090da70 con 0x555710a212d0 2023-07-14T19:24:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55571090d770 con 0x555710a22850 2023-07-14T19:24:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.620+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557107f18d0 con 0x555710a21c20 2023-07-14T19:24:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ec0fbb80 10 client.0 _ll_register_callbacks cb 0x55571090b710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ddcff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a2ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ddcff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a2ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51030/0 (socket says 172.21.15.53:51030) 2023-07-14T19:24:33.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ddcff700 1 -- 172.21.15.53:0/3905982281 learned_addr learned my addr 172.21.15.53:0/3905982281 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a2b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48de500700 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a32280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ddcff700 1 -- 172.21.15.53:0/3905982281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 msgr2=0x555710a2b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ddcff700 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a2b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ddcff700 1 -- 172.21.15.53:0/3905982281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 msgr2=0x555710a32280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ddcff700 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a32280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ddcff700 1 -- 172.21.15.53:0/3905982281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d4011ad0 con 0x555710a212d0 2023-07-14T19:24:33.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ddcff700 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a2ad00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f48d40097d0 tx=0x7f48d401f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32d9c49822c84e9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48c17fa700 1 -- 172.21.15.53:0/3905982281 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d402c070 con 0x555710a212d0 2023-07-14T19:24:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48c17fa700 1 -- 172.21.15.53:0/3905982281 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f48d401fe40 con 0x555710a212d0 2023-07-14T19:24:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3905982281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5557108c74a0 con 0x555710a212d0 2023-07-14T19:24:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48c17fa700 1 -- 172.21.15.53:0/3905982281 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48d4011e50 con 0x555710a212d0 2023-07-14T19:24:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ec0fbb80 10 client.15036 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.621+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3905982281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555710784c10 con 0x555710a212d0 2023-07-14T19:24:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48c17fa700 1 -- 172.21.15.53:0/3905982281 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5557108c74a0 con 0x555710a212d0 2023-07-14T19:24:33.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48c17fa700 1 -- 172.21.15.53:0/3905982281 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f48d403e400 con 0x555710a212d0 2023-07-14T19:24:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48ec0fbb80 10 client.15036 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48ec0fbb80 10 client.15036 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T19:24:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48ec0fbb80 20 client.15036 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x555710a35f20 0x555710a352d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48ec0fbb80 4 client.15036 mds_command: new command op to 24461 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:24:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3905982281 --> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557109ebd50 con 0x555710a35f20 2023-07-14T19:24:33.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x555710a35f20 0x555710a352d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x555710a35f20 0x555710a352d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=980f41ca43160b8e server_cookie=1e92a7e6e2501ec9 in_seq=0 out_seq=0 2023-07-14T19:24:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.622+0000 7f48c17fa700 10 client.15036 ms_handle_connect on v2:172.21.15.70:6834/2384359054 2023-07-14T19:24:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.623+0000 7f48c17fa700 1 -- 172.21.15.53:0/3905982281 <== mds.2 v2:172.21.15.70:6834/2384359054 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5557109ebd50 con 0x555710a35f20 2023-07-14T19:24:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.623+0000 7f48c17fa700 10 client.15036 handle_command_reply: tid=0 2023-07-14T19:24:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.623+0000 7f48ec0fbb80 1 client.15036 shutdown 2023-07-14T19:24:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.623+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3905982281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 msgr2=0x555710a2ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.623+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a2ad00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f48d40097d0 tx=0x7f48d401f6a0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.638 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3905982281 shutdown_connections 2023-07-14T19:24:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x555710a35f20 0x555710a352d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a32280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a2ad00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3905982281 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a2b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48c17fa700 0 client.15036 ms_handle_reset on v2:172.21.15.70:6834/2384359054 2023-07-14T19:24:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3905982281 >> 172.21.15.53:0/3905982281 conn(0x555710a12630 msgr2=0x555710a16600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.639 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3905982281 shutdown_connections 2023-07-14T19:24:33.640 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3905982281 wait complete. 2023-07-14T19:24:33.640 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.624+0000 7f48ec0fbb80 20 client.15036 trim_cache size 0 max 16384 2023-07-14T19:24:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ec0fbb80 1 Processor -- start 2023-07-14T19:24:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ec0fbb80 1 -- start start 2023-07-14T19:24:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a25680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a25bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.646 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a32240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x555710a0f930 con 0x555710a212d0 2023-07-14T19:24:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x555710a22170 con 0x555710a22850 2023-07-14T19:24:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557108c74a0 con 0x555710a21c20 2023-07-14T19:24:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ddcff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a25680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ddcff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a25680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51038/0 (socket says 172.21.15.53:51038) 2023-07-14T19:24:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.641+0000 7f48ddcff700 1 -- 172.21.15.53:0/1313051535 learned_addr learned my addr 172.21.15.53:0/1313051535 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ddcff700 1 -- 172.21.15.53:0/1313051535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 msgr2=0x555710a25bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48de500700 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a32240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a25bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ddcff700 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a25bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ddcff700 1 -- 172.21.15.53:0/1313051535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 msgr2=0x555710a32240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ddcff700 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a32240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ddcff700 1 -- 172.21.15.53:0/1313051535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557107c2200 con 0x555710a212d0 2023-07-14T19:24:33.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48de500700 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a32240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ddcff700 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a25680 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009340 tx=0x7f48d40099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52558857117f4933 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ccff9700 1 -- 172.21.15.53:0/1313051535 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d402c070 con 0x555710a212d0 2023-07-14T19:24:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ccff9700 1 -- 172.21.15.53:0/1313051535 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d4039020 con 0x555710a212d0 2023-07-14T19:24:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ccff9700 1 -- 172.21.15.53:0/1313051535 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40132a0 con 0x555710a212d0 2023-07-14T19:24:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.642+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/1313051535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 msgr2=0x555710a25680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a25680 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009340 tx=0x7f48d40099e0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.649 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/1313051535 shutdown_connections 2023-07-14T19:24:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a22850 0x555710a32240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a212d0 0x555710a25680 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/1313051535 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a21c20 0x555710a25bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/1313051535 >> 172.21.15.53:0/1313051535 conn(0x555710a12630 msgr2=0x555710a352d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/1313051535 shutdown_connections 2023-07-14T19:24:33.650 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/1313051535 wait complete. 2023-07-14T19:24:33.651 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 -- start start 2023-07-14T19:24:33.651 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 Processor -- start 2023-07-14T19:24:33.651 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a212d0 0x555710a2db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.651 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a21c20 0x555710a2e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.651 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a22850 0x555710a30f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.651 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x555710a31470 con 0x555710a21c20 2023-07-14T19:24:33.651 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x555710a315e0 con 0x555710a212d0 2023-07-14T19:24:33.651 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x555710a317a0 con 0x555710a22850 2023-07-14T19:24:33.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48ec0fbb80 10 client.0 _ll_register_callbacks cb 0x7f48d0009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:33.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48dd4fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a21c20 0x555710a2e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48dd4fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a21c20 0x555710a2e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51046/0 (socket says 172.21.15.53:51046) 2023-07-14T19:24:33.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.643+0000 7f48dd4fe700 1 -- 172.21.15.53:0/3545811837 learned_addr learned my addr 172.21.15.53:0/3545811837 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48ddcff700 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a212d0 0x555710a2db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48de500700 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a22850 0x555710a30f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.652 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48dd4fe700 1 -- 172.21.15.53:0/3545811837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a22850 msgr2=0x555710a30f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a22850 0x555710a30f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48dd4fe700 1 -- 172.21.15.53:0/3545811837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a212d0 msgr2=0x555710a2db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a212d0 0x555710a2db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48dd4fe700 1 -- 172.21.15.53:0/3545811837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555710784c10 con 0x555710a21c20 2023-07-14T19:24:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48ddcff700 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a212d0 0x555710a2db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48dd4fe700 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a21c20 0x555710a2e050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f48c80058f0 tx=0x7f48c800bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e144ff256c8f693 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:33.654 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48c17fa700 1 -- 172.21.15.53:0/3545811837 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c8017040 con 0x555710a21c20 2023-07-14T19:24:33.654 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48c17fa700 1 -- 172.21.15.53:0/3545811837 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f48c801ce20 con 0x555710a21c20 2023-07-14T19:24:33.654 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3545811837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48d4016070 con 0x555710a21c20 2023-07-14T19:24:33.654 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48c17fa700 1 -- 172.21.15.53:0/3545811837 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48c8003680 con 0x555710a21c20 2023-07-14T19:24:33.654 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48ec0fbb80 10 client.15042 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:33.654 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3545811837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55571079a260 con 0x555710a21c20 2023-07-14T19:24:33.654 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.644+0000 7f48c17fa700 1 -- 172.21.15.53:0/3545811837 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d4016070 con 0x555710a21c20 2023-07-14T19:24:33.654 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48c17fa700 1 -- 172.21.15.53:0/3545811837 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f48c8011890 con 0x555710a21c20 2023-07-14T19:24:33.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48ec0fbb80 10 client.15042 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:33.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48ec0fbb80 10 client.15042 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T19:24:33.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48ec0fbb80 20 client.15042 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:33.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x555710a8f7b0 0x555710a8c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48ec0fbb80 4 client.15042 mds_command: new command op to 24461 tid=0[{"prefix": "status"}] 2023-07-14T19:24:33.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3545811837 --> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555710966e60 con 0x555710a8f7b0 2023-07-14T19:24:33.655 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48ddcff700 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x555710a8f7b0 0x555710a8c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.656 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48c17fa700 10 client.15042 ms_handle_connect on v2:172.21.15.70:6834/2384359054 2023-07-14T19:24:33.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48ddcff700 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x555710a8f7b0 0x555710a8c120 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed68feaddab06df5 server_cookie=a0b63a19346a4844 in_seq=0 out_seq=0 2023-07-14T19:24:33.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48c17fa700 1 -- 172.21.15.53:0/3545811837 <== mds.2 v2:172.21.15.70:6834/2384359054 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x555710966e60 con 0x555710a8f7b0 2023-07-14T19:24:33.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.645+0000 7f48c17fa700 10 client.15042 handle_command_reply: tid=0 2023-07-14T19:24:33.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 client.15042 shutdown 2023-07-14T19:24:33.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3545811837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a21c20 msgr2=0x555710a2e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.657 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a21c20 0x555710a2e050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f48c80058f0 tx=0x7f48c800bd10 comp rx=0 tx=0).stop 2023-07-14T19:24:33.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3545811837 shutdown_connections 2023-07-14T19:24:33.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x555710a8f7b0 0x555710a8c120 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555710a22850 0x555710a30f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48c17fa700 0 client.15042 ms_handle_reset on v2:172.21.15.70:6834/2384359054 2023-07-14T19:24:33.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555710a212d0 0x555710a2db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 --2- 172.21.15.53:0/3545811837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x555710a21c20 0x555710a2e050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3545811837 >> 172.21.15.53:0/3545811837 conn(0x555710a12630 msgr2=0x555710a25680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.658 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3545811837 shutdown_connections 2023-07-14T19:24:33.659 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 1 -- 172.21.15.53:0/3545811837 wait complete. 2023-07-14T19:24:33.659 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48ec0fbb80 20 client.15042 trim_cache size 0 max 16384 2023-07-14T19:24:33.659 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-07-14T19:24:33.659 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:24:33.660 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 2, 2023-07-14T19:24:33.660 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24461, 2023-07-14T19:24:33.660 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2023-07-14T19:24:33.660 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2023-07-14T19:24:33.660 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2023-07-14T19:24:33.660 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 51.361183228999998, 2023-07-14T19:24:33.660 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 23, 2023-07-14T19:24:33.661 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 80, 2023-07-14T19:24:33.661 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 80, 2023-07-14T19:24:33.661 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 77.293854178999993 2023-07-14T19:24:33.661 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-07-14T19:24:33.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 -- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48c40a11e0 msgr2=0x7f48c40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48c40a11e0 0x7f48c40a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f48d80ffb60 tx=0x7f48d4009260 comp rx=0 tx=0).stop 2023-07-14T19:24:33.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 -- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 msgr2=0x7f48d819ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 0x7f48d819ce10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f48d000d010 tx=0x7f48d000bc80 comp rx=0 tx=0).stop 2023-07-14T19:24:33.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 -- 172.21.15.53:0/1725230553 shutdown_connections 2023-07-14T19:24:33.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f48c40a11e0 0x7f48c40a3690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48d8100690 0x7f48d819ce10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48d80feb10 0x7f48d8069360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 --2- 172.21.15.53:0/1725230553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d80ffd00 0x7f48d819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.646+0000 7f48dff63700 1 -- 172.21.15.53:0/1725230553 >> 172.21.15.53:0/1725230553 conn(0x7f48d80fa130 msgr2=0x7f48d80fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.647+0000 7f48dff63700 1 -- 172.21.15.53:0/1725230553 shutdown_connections 2023-07-14T19:24:33.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.647+0000 7f48dff63700 1 -- 172.21.15.53:0/1725230553 wait complete. 2023-07-14T19:24:33.664 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T19:24:33.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:33 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1368425110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:33 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/4261154874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:33.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:33 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/558883822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:33.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:33 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1368425110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:33.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:33 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/4261154874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:33.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:33 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/558883822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:33.791 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3937189382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b480feb40 msgr2=0x7f6b480fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3937189382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b480feb40 0x7f6b480fef50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009990 tx=0x7f6b3c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3937189382 shutdown_connections 2023-07-14T19:24:33.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3937189382 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b481006c0 0x7f6b48106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3937189382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b480feb40 0x7f6b480fef50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3937189382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b480ffd30 0x7f6b48100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3937189382 >> 172.21.15.53:0/3937189382 conn(0x7f6b480fa160 msgr2=0x7f6b480fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3937189382 shutdown_connections 2023-07-14T19:24:33.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3937189382 wait complete. 2023-07-14T19:24:33.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 Processor -- start 2023-07-14T19:24:33.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.790+0000 7f6b4ee7d700 1 -- start start 2023-07-14T19:24:33.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4ee7d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b480feb40 0x7f6b481948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4ee7d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b480ffd30 0x7f6b48194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4ee7d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b481006c0 0x7f6b4819be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4ee7d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4819c420 con 0x7f6b481006c0 2023-07-14T19:24:33.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4ee7d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4819c560 con 0x7f6b480ffd30 2023-07-14T19:24:33.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4ee7d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4819c6f0 con 0x7f6b480feb40 2023-07-14T19:24:33.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4d41a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b481006c0 0x7f6b4819be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4d41a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b481006c0 0x7f6b4819be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51062/0 (socket says 172.21.15.53:51062) 2023-07-14T19:24:33.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4d41a700 1 -- 172.21.15.53:0/3634379888 learned_addr learned my addr 172.21.15.53:0/3634379888 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b480feb40 0x7f6b481948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b47fff700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b480ffd30 0x7f6b48194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4d41a700 1 -- 172.21.15.53:0/3634379888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b480feb40 msgr2=0x7f6b481948a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b480feb40 0x7f6b481948a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4d41a700 1 -- 172.21.15.53:0/3634379888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b480ffd30 msgr2=0x7f6b48194de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b480ffd30 0x7f6b48194de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4d41a700 1 -- 172.21.15.53:0/3634379888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c0095f0 con 0x7f6b481006c0 2023-07-14T19:24:33.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b481006c0 0x7f6b4819be50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000ada0 tx=0x7f6b4000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b45ffb700 1 -- 172.21.15.53:0/3634379888 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000f070 con 0x7f6b481006c0 2023-07-14T19:24:33.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b45ffb700 1 -- 172.21.15.53:0/3634379888 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b40007b30 con 0x7f6b481006c0 2023-07-14T19:24:33.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3634379888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b48068890 con 0x7f6b481006c0 2023-07-14T19:24:33.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.791+0000 7f6b45ffb700 1 -- 172.21.15.53:0/3634379888 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40005280 con 0x7f6b481006c0 2023-07-14T19:24:33.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.792+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3634379888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b48068d50 con 0x7f6b481006c0 2023-07-14T19:24:33.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.794+0000 7f6b45ffb700 1 -- 172.21.15.53:0/3634379888 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6b4000c8d0 con 0x7f6b481006c0 2023-07-14T19:24:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.794+0000 7f6b45ffb700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6b380a11e0 0x7f6b380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.794+0000 7f6b45ffb700 1 -- 172.21.15.53:0/3634379888 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6b400cbe20 con 0x7f6b481006c0 2023-07-14T19:24:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.794+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6b380a11e0 0x7f6b380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6b380a11e0 0x7f6b380a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c006010 tx=0x7f6b3c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b5b015b80 1 Processor -- start 2023-07-14T19:24:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b5b015b80 1 -- start start 2023-07-14T19:24:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cbf10 0x55d2499cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cd460 0x55d2499cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2498d0c60 con 0x55d2499cc860 2023-07-14T19:24:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2499239f0 con 0x55d2499cd460 2023-07-14T19:24:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d24991ce80 con 0x55d2499cbf10 2023-07-14T19:24:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.795+0000 7f6b47fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b47fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51068/0 (socket says 172.21.15.53:51068) 2023-07-14T19:24:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b47fff700 1 -- 172.21.15.53:0/1145521840 learned_addr learned my addr 172.21.15.53:0/1145521840 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cbf10 0x55d2499cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b47fff700 1 -- 172.21.15.53:0/1145521840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cbf10 msgr2=0x55d2499cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b47fff700 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cbf10 0x55d2499cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b47fff700 1 -- 172.21.15.53:0/1145521840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cd460 msgr2=0x55d2499cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cd460 0x55d2499cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b47fff700 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cd460 0x55d2499cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b47fff700 1 -- 172.21.15.53:0/1145521840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d24974ad90 con 0x55d2499cc860 2023-07-14T19:24:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cbf10 0x55d2499cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cd460 0x55d2499cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b47fff700 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499cce80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009d80 tx=0x7f6b34011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea07324d54e8ef2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b32ffd700 1 -- 172.21.15.53:0/1145521840 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34004230 con 0x55d2499cc860 2023-07-14T19:24:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b32ffd700 1 -- 172.21.15.53:0/1145521840 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b34004390 con 0x55d2499cc860 2023-07-14T19:24:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.796+0000 7f6b32ffd700 1 -- 172.21.15.53:0/1145521840 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34019880 con 0x55d2499cc860 2023-07-14T19:24:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 -- 172.21.15.53:0/1145521840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 msgr2=0x55d2499cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499cce80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009d80 tx=0x7f6b34011720 comp rx=0 tx=0).stop 2023-07-14T19:24:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 -- 172.21.15.53:0/1145521840 shutdown_connections 2023-07-14T19:24:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cd460 0x55d2499cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cbf10 0x55d2499cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/1145521840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499cce80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.807 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 -- 172.21.15.53:0/1145521840 >> 172.21.15.53:0/1145521840 conn(0x55d2499c1630 msgr2=0x55d2499c6980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.807 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 -- 172.21.15.53:0/1145521840 shutdown_connections 2023-07-14T19:24:33.807 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 -- 172.21.15.53:0/1145521840 wait complete. 2023-07-14T19:24:33.807 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 -- start start 2023-07-14T19:24:33.807 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.797+0000 7f6b5b015b80 1 Processor -- start 2023-07-14T19:24:33.807 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 0x55d2499d5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.807 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cc860 0x55d2499d5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.808 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cd460 0x55d2499dcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.808 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2498bca70 con 0x55d2499cd460 2023-07-14T19:24:33.808 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2498bc770 con 0x55d2499cbf10 2023-07-14T19:24:33.808 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2497a08d0 con 0x55d2499cc860 2023-07-14T19:24:33.808 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b5b015b80 10 client.0 _ll_register_callbacks cb 0x55d2498ba710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:33.808 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4d41a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cd460 0x55d2499dcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.808 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4d41a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cd460 0x55d2499dcef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51076/0 (socket says 172.21.15.53:51076) 2023-07-14T19:24:33.808 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4d41a700 1 -- 172.21.15.53:0/417040050 learned_addr learned my addr 172.21.15.53:0/417040050 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 0x55d2499d5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b47fff700 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cc860 0x55d2499d5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4d41a700 1 -- 172.21.15.53:0/417040050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cc860 msgr2=0x55d2499d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cc860 0x55d2499d5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4d41a700 1 -- 172.21.15.53:0/417040050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 msgr2=0x55d2499d5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 0x55d2499d5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4d41a700 1 -- 172.21.15.53:0/417040050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b340099e0 con 0x55d2499cd460 2023-07-14T19:24:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 0x55d2499d5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cd460 0x55d2499dcef0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6b40095810 tx=0x7f6b4009d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad59b51e039f73ad server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b27fff700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b27fff700 1 -- 172.21.15.53:0/417040050 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40099770 con 0x55d2499cd460 2023-07-14T19:24:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b27fff700 1 -- 172.21.15.53:0/417040050 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6b4009d120 con 0x55d2499cd460 2023-07-14T19:24:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.798+0000 7f6b5b015b80 1 -- 172.21.15.53:0/417040050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d2498764a0 con 0x55d2499cd460 2023-07-14T19:24:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b27fff700 1 -- 172.21.15.53:0/417040050 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b40098800 con 0x55d2499cd460 2023-07-14T19:24:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b5b015b80 10 client.15054 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b5b015b80 1 -- 172.21.15.53:0/417040050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d249733c10 con 0x55d2499cd460 2023-07-14T19:24:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b27fff700 1 -- 172.21.15.53:0/417040050 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d2498764a0 con 0x55d2499cd460 2023-07-14T19:24:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b27fff700 1 -- 172.21.15.53:0/417040050 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f6b4000cb80 con 0x55d2499cd460 2023-07-14T19:24:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b5b015b80 10 client.15054 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b5b015b80 10 client.15054 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:24:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b5b015b80 20 client.15054 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x55d2499e0b70 0x55d2499dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b5b015b80 4 client.15054 mds_command: new command op to 24491 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:24:33.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b5b015b80 1 -- 172.21.15.53:0/417040050 --> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d24999ad50 con 0x55d2499e0b70 2023-07-14T19:24:33.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.799+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x55d2499e0b70 0x55d2499dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.800+0000 7f6b27fff700 10 client.15054 ms_handle_connect on v2:172.21.15.70:6836/3730963239 2023-07-14T19:24:33.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.800+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x55d2499e0b70 0x55d2499dffb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2087f0157726c9bc server_cookie=1b563c2107b2a23b in_seq=0 out_seq=0 2023-07-14T19:24:33.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.800+0000 7f6b27fff700 1 -- 172.21.15.53:0/417040050 <== mds.1 v2:172.21.15.70:6836/3730963239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55d24999ad50 con 0x55d2499e0b70 2023-07-14T19:24:33.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.800+0000 7f6b27fff700 10 client.15054 handle_command_reply: tid=0 2023-07-14T19:24:33.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.800+0000 7f6b5b015b80 1 client.15054 shutdown 2023-07-14T19:24:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.800+0000 7f6b5b015b80 1 -- 172.21.15.53:0/417040050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cd460 msgr2=0x55d2499dcef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.800+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cd460 0x55d2499dcef0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6b40095810 tx=0x7f6b4009d3c0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b5b015b80 1 -- 172.21.15.53:0/417040050 shutdown_connections 2023-07-14T19:24:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x55d2499e0b70 0x55d2499dffb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cd460 0x55d2499dcef0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 0x55d2499d5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/417040050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cc860 0x55d2499d5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b5b015b80 1 -- 172.21.15.53:0/417040050 >> 172.21.15.53:0/417040050 conn(0x55d2499c1630 msgr2=0x55d2499c5600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.814 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b27fff700 0 client.15054 ms_handle_reset on v2:172.21.15.70:6836/3730963239 2023-07-14T19:24:33.814 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b5b015b80 1 -- 172.21.15.53:0/417040050 shutdown_connections 2023-07-14T19:24:33.814 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b5b015b80 1 -- 172.21.15.53:0/417040050 wait complete. 2023-07-14T19:24:33.814 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.801+0000 7f6b5b015b80 20 client.15054 trim_cache size 0 max 16384 2023-07-14T19:24:33.820 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.818+0000 7f6b5b015b80 1 Processor -- start 2023-07-14T19:24:33.820 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.818+0000 7f6b5b015b80 1 -- start start 2023-07-14T19:24:33.821 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.818+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cbf10 0x55d2499d3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.821 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.818+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.821 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.818+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 0x55d2499dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.821 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.818+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2499c36b0 con 0x55d2499cbf10 2023-07-14T19:24:33.821 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.818+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2499deb50 con 0x55d2499cc860 2023-07-14T19:24:33.821 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.818+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2499d6b60 con 0x55d2499cd460 2023-07-14T19:24:33.821 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4cc19700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cbf10 0x55d2499d3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4cc19700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cbf10 0x55d2499d3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51080/0 (socket says 172.21.15.53:51080) 2023-07-14T19:24:33.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4cc19700 1 -- 172.21.15.53:0/79079417 learned_addr learned my addr 172.21.15.53:0/79079417 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/79079417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 0x55d2499dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b47fff700 1 --2- 172.21.15.53:0/79079417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4cc19700 1 -- 172.21.15.53:0/79079417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 msgr2=0x55d2499dffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.822 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/79079417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 0x55d2499dffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.823 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4cc19700 1 -- 172.21.15.53:0/79079417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cc860 msgr2=0x55d2499dceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.823 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/79079417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.823 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4cc19700 1 -- 172.21.15.53:0/79079417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d249771200 con 0x55d2499cbf10 2023-07-14T19:24:33.823 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/79079417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cbf10 0x55d2499d3820 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00b2e0 tx=0x7f6b3c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c49c7aaae82002a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.824 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b337fe700 1 -- 172.21.15.53:0/79079417 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c037070 con 0x55d2499cbf10 2023-07-14T19:24:33.824 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b337fe700 1 -- 172.21.15.53:0/79079417 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b3c01e040 con 0x55d2499cbf10 2023-07-14T19:24:33.824 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.819+0000 7f6b337fe700 1 -- 172.21.15.53:0/79079417 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c024410 con 0x55d2499cbf10 2023-07-14T19:24:33.827 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 -- 172.21.15.53:0/79079417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cbf10 msgr2=0x55d2499d3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.827 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/79079417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cbf10 0x55d2499d3820 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00b2e0 tx=0x7f6b3c020000 comp rx=0 tx=0).stop 2023-07-14T19:24:33.827 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 -- 172.21.15.53:0/79079417 shutdown_connections 2023-07-14T19:24:33.827 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/79079417 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 0x55d2499dffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.828 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/79079417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cbf10 0x55d2499d3820 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.828 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/79079417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.828 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 -- 172.21.15.53:0/79079417 >> 172.21.15.53:0/79079417 conn(0x55d2499c1630 msgr2=0x55d2499d4620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.828 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 -- 172.21.15.53:0/79079417 shutdown_connections 2023-07-14T19:24:33.828 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 -- 172.21.15.53:0/79079417 wait complete. 2023-07-14T19:24:33.828 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 -- start start 2023-07-14T19:24:33.828 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.820+0000 7f6b5b015b80 1 Processor -- start 2023-07-14T19:24:33.828 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 0x55d2499c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b5b015b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 0x55d2499c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2499c6c30 con 0x55d2499cc860 2023-07-14T19:24:33.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2499c6da0 con 0x55d2499cbf10 2023-07-14T19:24:33.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b5b015b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d249a347d0 con 0x55d2499cd460 2023-07-14T19:24:33.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b5b015b80 10 client.0 _ll_register_callbacks cb 0x7f6b40004ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:33.829 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b47fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b47fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51088/0 (socket says 172.21.15.53:51088) 2023-07-14T19:24:33.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b47fff700 1 -- 172.21.15.53:0/3448160962 learned_addr learned my addr 172.21.15.53:0/3448160962 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b4d41a700 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 0x55d2499c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 0x55d2499c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b47fff700 1 -- 172.21.15.53:0/3448160962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 msgr2=0x55d2499c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b47fff700 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 0x55d2499c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b47fff700 1 -- 172.21.15.53:0/3448160962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 msgr2=0x55d2499c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.830 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b47fff700 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 0x55d2499c60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.831 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b47fff700 1 -- 172.21.15.53:0/3448160962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c02c040 con 0x55d2499cc860 2023-07-14T19:24:33.831 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b47fff700 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6b340097c0 tx=0x7f6b34005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e24d489af698578 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.831 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b27fff700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:33.831 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b27fff700 1 -- 172.21.15.53:0/3448160962 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34004990 con 0x55d2499cc860 2023-07-14T19:24:33.831 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b27fff700 1 -- 172.21.15.53:0/3448160962 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6b34004af0 con 0x55d2499cc860 2023-07-14T19:24:33.831 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b5b015b80 1 -- 172.21.15.53:0/3448160962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b34004390 con 0x55d2499cc860 2023-07-14T19:24:33.831 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.821+0000 7f6b27fff700 1 -- 172.21.15.53:0/3448160962 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b34004cb0 con 0x55d2499cc860 2023-07-14T19:24:33.831 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b5b015b80 10 client.15060 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:33.832 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b5b015b80 1 -- 172.21.15.53:0/3448160962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d249749260 con 0x55d2499cc860 2023-07-14T19:24:33.832 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b27fff700 1 -- 172.21.15.53:0/3448160962 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b34004390 con 0x55d2499cc860 2023-07-14T19:24:33.832 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b27fff700 1 -- 172.21.15.53:0/3448160962 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f6b34021070 con 0x55d2499cc860 2023-07-14T19:24:33.832 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b5b015b80 10 client.15060 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:33.832 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b5b015b80 10 client.15060 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:24:33.832 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b5b015b80 20 client.15060 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:33.832 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x55d249a393a0 0x55d249a35cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b5b015b80 4 client.15060 mds_command: new command op to 24491 tid=0[{"prefix": "status"}] 2023-07-14T19:24:33.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b5b015b80 1 -- 172.21.15.53:0/3448160962 --> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d249915e60 con 0x55d249a393a0 2023-07-14T19:24:33.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x55d249a393a0 0x55d249a35cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b4cc19700 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x55d249a393a0 0x55d249a35cd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=17a30324bfbb3198 server_cookie=bd19e1ad67f9d3d0 in_seq=0 out_seq=0 2023-07-14T19:24:33.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.822+0000 7f6b27fff700 10 client.15060 ms_handle_connect on v2:172.21.15.70:6836/3730963239 2023-07-14T19:24:33.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b27fff700 1 -- 172.21.15.53:0/3448160962 <== mds.1 v2:172.21.15.70:6836/3730963239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55d249915e60 con 0x55d249a393a0 2023-07-14T19:24:33.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b27fff700 10 client.15060 handle_command_reply: tid=0 2023-07-14T19:24:33.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 client.15060 shutdown 2023-07-14T19:24:33.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 -- 172.21.15.53:0/3448160962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 msgr2=0x55d2499dceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6b340097c0 tx=0x7f6b34005800 comp rx=0 tx=0).stop 2023-07-14T19:24:33.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 -- 172.21.15.53:0/3448160962 shutdown_connections 2023-07-14T19:24:33.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x55d249a393a0 0x55d249a35cd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55d2499cd460 0x55d2499c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b27fff700 0 client.15060 ms_handle_reset on v2:172.21.15.70:6836/3730963239 2023-07-14T19:24:33.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d2499cbf10 0x55d2499c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 --2- 172.21.15.53:0/3448160962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2499cc860 0x55d2499dceb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 -- 172.21.15.53:0/3448160962 >> 172.21.15.53:0/3448160962 conn(0x55d2499c1630 msgr2=0x55d2499dffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 -- 172.21.15.53:0/3448160962 shutdown_connections 2023-07-14T19:24:33.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 1 -- 172.21.15.53:0/3448160962 wait complete. 2023-07-14T19:24:33.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.823+0000 7f6b5b015b80 20 client.15060 trim_cache size 0 max 16384 2023-07-14T19:24:33.836 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-07-14T19:24:33.836 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:24:33.836 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 1, 2023-07-14T19:24:33.836 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24491, 2023-07-14T19:24:33.836 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2023-07-14T19:24:33.836 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2023-07-14T19:24:33.837 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2023-07-14T19:24:33.837 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 52.357116032999997, 2023-07-14T19:24:33.837 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 23, 2023-07-14T19:24:33.837 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 80, 2023-07-14T19:24:33.837 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 80, 2023-07-14T19:24:33.837 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 70.996886982999996 2023-07-14T19:24:33.837 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-07-14T19:24:33.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6b380a11e0 msgr2=0x7f6b380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6b380a11e0 0x7f6b380a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c006010 tx=0x7f6b3c005960 comp rx=0 tx=0).stop 2023-07-14T19:24:33.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b481006c0 msgr2=0x7f6b4819be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b481006c0 0x7f6b4819be50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000ada0 tx=0x7f6b4000c330 comp rx=0 tx=0).stop 2023-07-14T19:24:33.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3634379888 shutdown_connections 2023-07-14T19:24:33.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6b380a11e0 0x7f6b380a3690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b481006c0 0x7f6b4819be50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6b480feb40 0x7f6b481948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 --2- 172.21.15.53:0/3634379888 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6b480ffd30 0x7f6b48194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3634379888 >> 172.21.15.53:0/3634379888 conn(0x7f6b480fa160 msgr2=0x7f6b48103270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3634379888 shutdown_connections 2023-07-14T19:24:33.840 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.824+0000 7f6b4ee7d700 1 -- 172.21.15.53:0/3634379888 wait complete. 2023-07-14T19:24:33.840 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T19:24:33.966 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 -- 172.21.15.53:0/815074652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10100ffd00 msgr2=0x7f1010100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.967 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 --2- 172.21.15.53:0/815074652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10100ffd00 0x7f1010100150 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1000009990 tx=0x7f1000009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.967 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 -- 172.21.15.53:0/815074652 shutdown_connections 2023-07-14T19:24:33.967 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 --2- 172.21.15.53:0/815074652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1010100690 0x7f1010106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.967 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 --2- 172.21.15.53:0/815074652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10100feb10 0x7f10100fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.967 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 --2- 172.21.15.53:0/815074652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10100ffd00 0x7f1010100150 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.967 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 -- 172.21.15.53:0/815074652 >> 172.21.15.53:0/815074652 conn(0x7f10100fa130 msgr2=0x7f10100fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.968 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 -- 172.21.15.53:0/815074652 shutdown_connections 2023-07-14T19:24:33.968 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 -- 172.21.15.53:0/815074652 wait complete. 2023-07-14T19:24:33.969 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.965+0000 7f10154ea700 1 Processor -- start 2023-07-14T19:24:33.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f10154ea700 1 -- start start 2023-07-14T19:24:33.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f10154ea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10100feb10 0x7f1010197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f10154ea700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10100ffd00 0x7f10101980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f10154ea700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1010100690 0x7f101019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f10154ea700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101019f700 con 0x7f1010100690 2023-07-14T19:24:33.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f10154ea700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101019f840 con 0x7f10100ffd00 2023-07-14T19:24:33.970 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f10154ea700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101019f9d0 con 0x7f10100feb10 2023-07-14T19:24:33.971 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1010100690 0x7f101019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.971 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1010100690 0x7f101019f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51104/0 (socket says 172.21.15.53:51104) 2023-07-14T19:24:33.971 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100f7fe700 1 -- 172.21.15.53:0/2798284654 learned_addr learned my addr 172.21.15.53:0/2798284654 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.971 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100e7fc700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10100ffd00 0x7f10101980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.971 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100f7fe700 1 -- 172.21.15.53:0/2798284654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10100feb10 msgr2=0x7f1010197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100effd700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10100feb10 0x7f1010197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100f7fe700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10100feb10 0x7f1010197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100f7fe700 1 -- 172.21.15.53:0/2798284654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10100ffd00 msgr2=0x7f10101980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100f7fe700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10100ffd00 0x7f10101980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100f7fe700 1 -- 172.21.15.53:0/2798284654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10000095f0 con 0x7f1010100690 2023-07-14T19:24:33.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.966+0000 7f100effd700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10100feb10 0x7f1010197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:33.972 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.967+0000 7f100f7fe700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1010100690 0x7f101019f130 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1008007ee0 tx=0x7f100800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.973 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.967+0000 7f0ff7fff700 1 -- 172.21.15.53:0/2798284654 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100800ddd0 con 0x7f1010100690 2023-07-14T19:24:33.973 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.967+0000 7f0ff7fff700 1 -- 172.21.15.53:0/2798284654 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f100800f040 con 0x7f1010100690 2023-07-14T19:24:33.973 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.967+0000 7f10154ea700 1 -- 172.21.15.53:0/2798284654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101019fcb0 con 0x7f1010100690 2023-07-14T19:24:33.973 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.967+0000 7f0ff7fff700 1 -- 172.21.15.53:0/2798284654 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080048e0 con 0x7f1010100690 2023-07-14T19:24:33.973 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.967+0000 7f10154ea700 1 -- 172.21.15.53:0/2798284654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10101a01c0 con 0x7f1010100690 2023-07-14T19:24:33.978 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.969+0000 7f0ff7fff700 1 -- 172.21.15.53:0/2798284654 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1008003a00 con 0x7f1010100690 2023-07-14T19:24:33.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.969+0000 7f0ff7fff700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0ff80a1260 0x7f0ff80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.970+0000 7f0ff7fff700 1 -- 172.21.15.53:0/2798284654 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f10080cc2b0 con 0x7f1010100690 2023-07-14T19:24:33.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.970+0000 7f100effd700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0ff80a1260 0x7f0ff80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.970+0000 7f100effd700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0ff80a1260 0x7f0ff80a3710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1004005fd0 tx=0x7f1004005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.970+0000 7f1021682b80 1 Processor -- start 2023-07-14T19:24:33.979 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.970+0000 7f1021682b80 1 -- start start 2023-07-14T19:24:33.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.970+0000 7f1021682b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3335f10 0x5565e3336320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f1021682b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 0x5565e3336e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f1021682b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3337460 0x5565e33378d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f1021682b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e323ac60 con 0x5565e3337460 2023-07-14T19:24:33.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f1021682b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e328d9f0 con 0x5565e3336860 2023-07-14T19:24:33.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f1021682b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e3286e80 con 0x5565e3335f10 2023-07-14T19:24:33.980 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3337460 0x5565e33378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100f7fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3337460 0x5565e33378d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51112/0 (socket says 172.21.15.53:51112) 2023-07-14T19:24:33.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100f7fe700 1 -- 172.21.15.53:0/3186378565 learned_addr learned my addr 172.21.15.53:0/3186378565 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100effd700 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3335f10 0x5565e3336320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100e7fc700 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 0x5565e3336e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100f7fe700 1 -- 172.21.15.53:0/3186378565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3335f10 msgr2=0x5565e3336320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100f7fe700 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3335f10 0x5565e3336320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.981 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100f7fe700 1 -- 172.21.15.53:0/3186378565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 msgr2=0x5565e3336e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100f7fe700 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 0x5565e3336e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100f7fe700 1 -- 172.21.15.53:0/3186378565 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565e30b4d90 con 0x5565e3337460 2023-07-14T19:24:33.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100e7fc700 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 0x5565e3336e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:33.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f100f7fe700 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3337460 0x5565e33378d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1008096080 tx=0x7f1008099460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794cff4d76e839a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f0ff57fa700 1 -- 172.21.15.53:0/3186378565 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008099610 con 0x5565e3337460 2023-07-14T19:24:33.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f0ff57fa700 1 -- 172.21.15.53:0/3186378565 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f100809d640 con 0x5565e3337460 2023-07-14T19:24:33.982 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.971+0000 7f0ff57fa700 1 -- 172.21.15.53:0/3186378565 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080840b0 con 0x5565e3337460 2023-07-14T19:24:33.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 -- 172.21.15.53:0/3186378565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3337460 msgr2=0x5565e33378d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3337460 0x5565e33378d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1008096080 tx=0x7f1008099460 comp rx=0 tx=0).stop 2023-07-14T19:24:33.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 -- 172.21.15.53:0/3186378565 shutdown_connections 2023-07-14T19:24:33.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3337460 0x5565e33378d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3335f10 0x5565e3336320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.983 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 --2- 172.21.15.53:0/3186378565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 0x5565e3336e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 -- 172.21.15.53:0/3186378565 >> 172.21.15.53:0/3186378565 conn(0x5565e332b630 msgr2=0x5565e3330980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 -- 172.21.15.53:0/3186378565 shutdown_connections 2023-07-14T19:24:33.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 -- 172.21.15.53:0/3186378565 wait complete. 2023-07-14T19:24:33.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 -- start start 2023-07-14T19:24:33.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.972+0000 7f1021682b80 1 Processor -- start 2023-07-14T19:24:33.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f1021682b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e333f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.984 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f1021682b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3336860 0x5565e333fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f1021682b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3337460 0x5565e3346ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f1021682b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e3226a70 con 0x5565e3335f10 2023-07-14T19:24:33.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f1021682b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e3226770 con 0x5565e3337460 2023-07-14T19:24:33.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f1021682b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e310a8d0 con 0x5565e3336860 2023-07-14T19:24:33.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f1021682b80 10 client.0 _ll_register_callbacks cb 0x5565e3224710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:33.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e333f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.985 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e333f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51114/0 (socket says 172.21.15.53:51114) 2023-07-14T19:24:33.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100effd700 1 -- 172.21.15.53:0/1998263778 learned_addr learned my addr 172.21.15.53:0/1998263778 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100f7fe700 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3337460 0x5565e3346ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100e7fc700 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3336860 0x5565e333fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100effd700 1 -- 172.21.15.53:0/1998263778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3336860 msgr2=0x5565e333fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100effd700 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3336860 0x5565e333fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100effd700 1 -- 172.21.15.53:0/1998263778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3337460 msgr2=0x5565e3346ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100effd700 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3337460 0x5565e3346ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100effd700 1 -- 172.21.15.53:0/1998263778 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10101a01c0 con 0x5565e3335f10 2023-07-14T19:24:33.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100f7fe700 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3337460 0x5565e3346ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:33.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100e7fc700 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3336860 0x5565e333fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:33.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f100effd700 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e333f940 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f10040117f0 tx=0x7f1004004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31321028d6f9db1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f0ff2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:33.987 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/1998263778 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004004a30 con 0x5565e3335f10 2023-07-14T19:24:33.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/1998263778 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1004004b90 con 0x5565e3335f10 2023-07-14T19:24:33.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f1021682b80 1 -- 172.21.15.53:0/1998263778 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5565e31e04a0 con 0x5565e3335f10 2023-07-14T19:24:33.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.973+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/1998263778 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1004004d90 con 0x5565e3335f10 2023-07-14T19:24:33.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f1021682b80 10 client.15072 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:33.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f1021682b80 1 -- 172.21.15.53:0/1998263778 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5565e309dc10 con 0x5565e3335f10 2023-07-14T19:24:33.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/1998263778 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5565e31e04a0 con 0x5565e3335f10 2023-07-14T19:24:33.988 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/1998263778 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f1004025030 con 0x5565e3335f10 2023-07-14T19:24:33.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f1021682b80 10 client.15072 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:33.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f1021682b80 10 client.15072 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:24:33.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f1021682b80 20 client.15072 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:33.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f1021682b80 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x5565e334ac10 0x5565e3349fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f1021682b80 4 client.15072 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:24:33.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f1021682b80 1 -- 172.21.15.53:0/1998263778 --> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5565e3304d50 con 0x5565e334ac10 2023-07-14T19:24:33.989 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.974+0000 7f100e7fc700 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x5565e334ac10 0x5565e3349fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f0ff2ff5700 10 client.15072 ms_handle_connect on v2:172.21.15.70:6838/2496341573 2023-07-14T19:24:33.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f100e7fc700 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x5565e334ac10 0x5565e3349fc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab5173b7b3c51676 server_cookie=a08e1e8922b7a07b in_seq=0 out_seq=0 2023-07-14T19:24:33.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/1998263778 <== mds.0 v2:172.21.15.70:6838/2496341573 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5565e3304d50 con 0x5565e334ac10 2023-07-14T19:24:33.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f0ff2ff5700 10 client.15072 handle_command_reply: tid=0 2023-07-14T19:24:33.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 client.15072 shutdown 2023-07-14T19:24:33.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 -- 172.21.15.53:0/1998263778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 msgr2=0x5565e333f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:33.990 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e333f940 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f10040117f0 tx=0x7f1004004750 comp rx=0 tx=0).stop 2023-07-14T19:24:33.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 -- 172.21.15.53:0/1998263778 shutdown_connections 2023-07-14T19:24:33.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x5565e334ac10 0x5565e3349fc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3337460 0x5565e3346ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f0ff2ff5700 0 client.15072 ms_handle_reset on v2:172.21.15.70:6838/2496341573 2023-07-14T19:24:33.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e333f940 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 --2- 172.21.15.53:0/1998263778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3336860 0x5565e333fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.991 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 -- 172.21.15.53:0/1998263778 >> 172.21.15.53:0/1998263778 conn(0x5565e332b630 msgr2=0x5565e332f600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:33.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.975+0000 7f1021682b80 1 -- 172.21.15.53:0/1998263778 shutdown_connections 2023-07-14T19:24:33.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.976+0000 7f1021682b80 1 -- 172.21.15.53:0/1998263778 wait complete. 2023-07-14T19:24:33.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.976+0000 7f1021682b80 20 client.15072 trim_cache size 0 max 16384 2023-07-14T19:24:33.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f1021682b80 1 Processor -- start 2023-07-14T19:24:33.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f1021682b80 1 -- start start 2023-07-14T19:24:33.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f1021682b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3335f10 0x5565e3349fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f1021682b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3336860 0x5565e334a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f1021682b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3337460 0x5565e3346e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:33.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f1021682b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e3340b30 con 0x5565e3336860 2023-07-14T19:24:33.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f1021682b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e3230470 con 0x5565e3335f10 2023-07-14T19:24:33.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f1021682b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e3255790 con 0x5565e3337460 2023-07-14T19:24:33.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f100e7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3336860 0x5565e334a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f100e7fc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3336860 0x5565e334a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51124/0 (socket says 172.21.15.53:51124) 2023-07-14T19:24:33.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.993+0000 7f100e7fc700 1 -- 172.21.15.53:0/3257984166 learned_addr learned my addr 172.21.15.53:0/3257984166 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:33.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f100e7fc700 1 -- 172.21.15.53:0/3257984166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3337460 msgr2=0x5565e3346e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f100e7fc700 1 --2- 172.21.15.53:0/3257984166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3337460 0x5565e3346e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f100e7fc700 1 -- 172.21.15.53:0/3257984166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3335f10 msgr2=0x5565e3349fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:33.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f100effd700 1 --2- 172.21.15.53:0/3257984166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3335f10 0x5565e3349fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:33.997 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f100e7fc700 1 --2- 172.21.15.53:0/3257984166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3335f10 0x5565e3349fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:33.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f100e7fc700 1 -- 172.21.15.53:0/3257984166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565e30db200 con 0x5565e3336860 2023-07-14T19:24:33.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f100e7fc700 1 --2- 172.21.15.53:0/3257984166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3336860 0x5565e334a500 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1000015000 tx=0x7f1000005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121813708ce8843b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:33.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f0ff5ffb700 1 -- 172.21.15.53:0/3257984166 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100001d040 con 0x5565e3336860 2023-07-14T19:24:33.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f0ff5ffb700 1 -- 172.21.15.53:0/3257984166 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f100001e040 con 0x5565e3336860 2023-07-14T19:24:33.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.994+0000 7f0ff5ffb700 1 -- 172.21.15.53:0/3257984166 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100000b9d0 con 0x5565e3336860 2023-07-14T19:24:34.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 -- 172.21.15.53:0/3257984166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3336860 msgr2=0x5565e334a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:34.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 --2- 172.21.15.53:0/3257984166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3336860 0x5565e334a500 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1000015000 tx=0x7f1000005e80 comp rx=0 tx=0).stop 2023-07-14T19:24:34.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 -- 172.21.15.53:0/3257984166 shutdown_connections 2023-07-14T19:24:34.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 --2- 172.21.15.53:0/3257984166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3337460 0x5565e3346e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 --2- 172.21.15.53:0/3257984166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3335f10 0x5565e3349fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 --2- 172.21.15.53:0/3257984166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3336860 0x5565e334a500 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.002 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 -- 172.21.15.53:0/3257984166 >> 172.21.15.53:0/3257984166 conn(0x5565e332b630 msgr2=0x5565e333d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:34.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 -- 172.21.15.53:0/3257984166 shutdown_connections 2023-07-14T19:24:34.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 -- 172.21.15.53:0/3257984166 wait complete. 2023-07-14T19:24:34.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 -- start start 2023-07-14T19:24:34.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 Processor -- start 2023-07-14T19:24:34.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e3342230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:34.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 0x5565e3342770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:34.003 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3337460 0x5565e3345790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:34.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e3345cd0 con 0x5565e3335f10 2023-07-14T19:24:34.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e3345e40 con 0x5565e3336860 2023-07-14T19:24:34.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565e3346000 con 0x5565e3337460 2023-07-14T19:24:34.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f1021682b80 10 client.0 _ll_register_callbacks cb 0x7f100400ef60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:34.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f100effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e3342230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:34.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f100effd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e3342230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51128/0 (socket says 172.21.15.53:51128) 2023-07-14T19:24:34.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.995+0000 7f100effd700 1 -- 172.21.15.53:0/2935742146 learned_addr learned my addr 172.21.15.53:0/2935742146 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:34.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f100e7fc700 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 0x5565e3342770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:34.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f100f7fe700 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3337460 0x5565e3345790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:34.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f100effd700 1 -- 172.21.15.53:0/2935742146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3337460 msgr2=0x5565e3345790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:34.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f100effd700 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3337460 0x5565e3345790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f100effd700 1 -- 172.21.15.53:0/2935742146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 msgr2=0x5565e3342770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:34.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f100effd700 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 0x5565e3342770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f100effd700 1 -- 172.21.15.53:0/2935742146 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1010107880 con 0x5565e3335f10 2023-07-14T19:24:34.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f100effd700 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e3342230 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1004009a00 tx=0x7f1004010aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7bd1d5782c3d44 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:34.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f0ff2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:34.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/2935742146 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004005240 con 0x5565e3335f10 2023-07-14T19:24:34.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f1021682b80 1 -- 172.21.15.53:0/2935742146 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f100809d640 con 0x5565e3335f10 2023-07-14T19:24:34.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/2935742146 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1004010820 con 0x5565e3335f10 2023-07-14T19:24:34.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/2935742146 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10040053a0 con 0x5565e3335f10 2023-07-14T19:24:34.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f1021682b80 10 client.15078 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:34.007 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f1021682b80 1 -- 172.21.15.53:0/2935742146 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5565e30b3260 con 0x5565e3335f10 2023-07-14T19:24:34.007 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.996+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/2935742146 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f100809d640 con 0x5565e3335f10 2023-07-14T19:24:34.007 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/2935742146 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f10040118d0 con 0x5565e3335f10 2023-07-14T19:24:34.007 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f1021682b80 10 client.15078 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:34.007 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f1021682b80 10 client.15078 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:24:34.007 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f1021682b80 20 client.15078 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:34.007 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f1021682b80 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x5565e33a41b0 0x5565e33a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f1021682b80 4 client.15078 mds_command: new command op to 24527 tid=0[{"prefix": "status"}] 2023-07-14T19:24:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f1021682b80 1 -- 172.21.15.53:0/2935742146 --> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5565e327fe60 con 0x5565e33a41b0 2023-07-14T19:24:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f100e7fc700 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x5565e33a41b0 0x5565e33a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f0ff2ff5700 10 client.15078 ms_handle_connect on v2:172.21.15.70:6838/2496341573 2023-07-14T19:24:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f100e7fc700 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x5565e33a41b0 0x5565e33a0b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7f9a68f6e69647a server_cookie=51a9c6e17bdc8af in_seq=0 out_seq=0 2023-07-14T19:24:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f0ff2ff5700 1 -- 172.21.15.53:0/2935742146 <== mds.0 v2:172.21.15.70:6838/2496341573 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5565e327fe60 con 0x5565e33a41b0 2023-07-14T19:24:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f0ff2ff5700 10 client.15078 handle_command_reply: tid=0 2023-07-14T19:24:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.997+0000 7f1021682b80 1 client.15078 shutdown 2023-07-14T19:24:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 -- 172.21.15.53:0/2935742146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 msgr2=0x5565e3342230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e3342230 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1004009a00 tx=0x7f1004010aa0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 -- 172.21.15.53:0/2935742146 shutdown_connections 2023-07-14T19:24:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x5565e33a41b0 0x5565e33a0b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5565e3337460 0x5565e3345790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f0ff2ff5700 0 client.15078 ms_handle_reset on v2:172.21.15.70:6838/2496341573 2023-07-14T19:24:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5565e3335f10 0x5565e3342230 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 --2- 172.21.15.53:0/2935742146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5565e3336860 0x5565e3342770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 -- 172.21.15.53:0/2935742146 >> 172.21.15.53:0/2935742146 conn(0x5565e332b630 msgr2=0x5565e3349fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 -- 172.21.15.53:0/2935742146 shutdown_connections 2023-07-14T19:24:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 1 -- 172.21.15.53:0/2935742146 wait complete. 2023-07-14T19:24:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f1021682b80 20 client.15078 trim_cache size 0 max 16384 2023-07-14T19:24:34.011 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-07-14T19:24:34.011 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:24:34.011 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2023-07-14T19:24:34.011 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24527, 2023-07-14T19:24:34.011 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2023-07-14T19:24:34.012 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2023-07-14T19:24:34.012 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2023-07-14T19:24:34.012 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 55.551594366000003, 2023-07-14T19:24:34.012 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 23, 2023-07-14T19:24:34.012 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 80, 2023-07-14T19:24:34.012 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 80, 2023-07-14T19:24:34.012 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 62.220103666 2023-07-14T19:24:34.013 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-07-14T19:24:34.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 -- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0ff80a1260 msgr2=0x7f0ff80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:34.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0ff80a1260 0x7f0ff80a3710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1004005fd0 tx=0x7f1004005f00 comp rx=0 tx=0).stop 2023-07-14T19:24:34.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 -- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1010100690 msgr2=0x7f101019f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:34.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1010100690 0x7f101019f130 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1008007ee0 tx=0x7f100800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 -- 172.21.15.53:0/2798284654 shutdown_connections 2023-07-14T19:24:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f0ff80a1260 0x7f0ff80a3710 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1010100690 0x7f101019f130 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f10100feb10 0x7f1010197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 --2- 172.21.15.53:0/2798284654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10100ffd00 0x7f10101980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 -- 172.21.15.53:0/2798284654 >> 172.21.15.53:0/2798284654 conn(0x7f10100fa130 msgr2=0x7f10100677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 -- 172.21.15.53:0/2798284654 shutdown_connections 2023-07-14T19:24:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:33.998+0000 7f10154ea700 1 -- 172.21.15.53:0/2798284654 wait complete. 2023-07-14T19:24:34.016 INFO:tasks.cephadm:Setup complete, yielding 2023-07-14T19:24:34.016 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T19:24:34.026 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi053.front.sepia.ceph.com 2023-07-14T19:24:34.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- bash -c 'ceph orch status' 2023-07-14T19:24:34.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:34 smithi053 ceph-mon[96565]: pgmap v276: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:24:34.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:34 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2855393828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:34.582 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:34.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:34 smithi070 ceph-mon[98015]: pgmap v276: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:24:34.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:34 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2855393828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:34.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:34 smithi104 ceph-mon[98435]: pgmap v276: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:24:34.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:34 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2855393828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:35.608 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:35 smithi053 ceph-mon[96565]: pgmap v277: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:24:35.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:35 smithi070 ceph-mon[98015]: pgmap v277: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:24:35.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.732+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/3457998220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a98100740 msgr2=0x7f3a98107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:35.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.732+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/3457998220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a98100740 0x7f3a98107000 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000b0a0 tx=0x7f3a9000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/3457998220 shutdown_connections 2023-07-14T19:24:35.735 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/3457998220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a98100740 0x7f3a98107000 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/3457998220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a980febb0 0x7f3a980fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/3457998220 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a980ffdb0 0x7f3a98100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/3457998220 >> 172.21.15.53:0/3457998220 conn(0x7f3a980fa1e0 msgr2=0x7f3a980fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:35.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/3457998220 shutdown_connections 2023-07-14T19:24:35.736 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/3457998220 wait complete. 2023-07-14T19:24:35.738 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 Processor -- start 2023-07-14T19:24:35.738 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 -- start start 2023-07-14T19:24:35.738 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.733+0000 7f3a9ce1d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a980febb0 0x7f3a98197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:35.738 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a9ce1d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a980ffdb0 0x7f3a981981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:35.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a9ce1d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a98100740 0x7f3a9819f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:35.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a9ce1d700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9819f810 con 0x7f3a980ffdb0 2023-07-14T19:24:35.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a9ce1d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9819f950 con 0x7f3a98100740 2023-07-14T19:24:35.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a9ce1d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9819fb10 con 0x7f3a980febb0 2023-07-14T19:24:35.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a95d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a980ffdb0 0x7f3a981981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:35.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a95d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a980ffdb0 0x7f3a981981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51144/0 (socket says 172.21.15.53:51144) 2023-07-14T19:24:35.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a95d9b700 1 -- 172.21.15.53:0/1897761668 learned_addr learned my addr 172.21.15.53:0/1897761668 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:35.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a9659c700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a980febb0 0x7f3a98197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:35.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a96d9d700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a98100740 0x7f3a9819f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:35.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a95d9b700 1 -- 172.21.15.53:0/1897761668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a980febb0 msgr2=0x7f3a98197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:35.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a95d9b700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a980febb0 0x7f3a98197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a95d9b700 1 -- 172.21.15.53:0/1897761668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a98100740 msgr2=0x7f3a9819f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:35.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a95d9b700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a98100740 0x7f3a9819f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a95d9b700 1 -- 172.21.15.53:0/1897761668 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a90009d00 con 0x7f3a980ffdb0 2023-07-14T19:24:35.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a9659c700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a980febb0 0x7f3a98197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:35.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a96d9d700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a98100740 0x7f3a9819f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:35.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a95d9b700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a980ffdb0 0x7f3a981981d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00b530 tx=0x7f3a8c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:35.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a877fe700 1 -- 172.21.15.53:0/1897761668 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c011830 con 0x7f3a980ffdb0 2023-07-14T19:24:35.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9819fdf0 con 0x7f3a980ffdb0 2023-07-14T19:24:35.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a877fe700 1 -- 172.21.15.53:0/1897761668 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a8c011e50 con 0x7f3a980ffdb0 2023-07-14T19:24:35.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a877fe700 1 -- 172.21.15.53:0/1897761668 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c00f2e0 con 0x7f3a980ffdb0 2023-07-14T19:24:35.742 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.734+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a981a02f0 con 0x7f3a980ffdb0 2023-07-14T19:24:35.742 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.735+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9810a3f0 con 0x7f3a980ffdb0 2023-07-14T19:24:35.742 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.736+0000 7f3a877fe700 1 -- 172.21.15.53:0/1897761668 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3a8c010580 con 0x7f3a980ffdb0 2023-07-14T19:24:35.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.737+0000 7f3a877fe700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3a800a11e0 0x7f3a800a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:35.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.737+0000 7f3a877fe700 1 -- 172.21.15.53:0/1897761668 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3a8c0cc850 con 0x7f3a980ffdb0 2023-07-14T19:24:35.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.737+0000 7f3a9659c700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3a800a11e0 0x7f3a800a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:35.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.737+0000 7f3a9659c700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3a800a11e0 0x7f3a800a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3a98198b70 tx=0x7f3a8800a2b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:35.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.740+0000 7f3a877fe700 1 -- 172.21.15.53:0/1897761668 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3a8c096100 con 0x7f3a980ffdb0 2023-07-14T19:24:35.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:35 smithi104 ceph-mon[98435]: pgmap v277: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:24:35.957 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.955+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a98068830 con 0x7f3a800a11e0 2023-07-14T19:24:35.958 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.956+0000 7f3a877fe700 1 -- 172.21.15.53:0/1897761668 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3a98068830 con 0x7f3a800a11e0 2023-07-14T19:24:35.958 INFO:teuthology.orchestra.run.smithi053.stdout:Backend: cephadm 2023-07-14T19:24:35.959 INFO:teuthology.orchestra.run.smithi053.stdout:Available: Yes 2023-07-14T19:24:35.959 INFO:teuthology.orchestra.run.smithi053.stdout:Paused: No 2023-07-14T19:24:35.960 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3a800a11e0 msgr2=0x7f3a800a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:35.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3a800a11e0 0x7f3a800a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3a98198b70 tx=0x7f3a8800a2b0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a980ffdb0 msgr2=0x7f3a981981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:35.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a980ffdb0 0x7f3a981981d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00b530 tx=0x7f3a8c00b8f0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 shutdown_connections 2023-07-14T19:24:35.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3a800a11e0 0x7f3a800a3690 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.961 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a98100740 0x7f3a9819f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a980febb0 0x7f3a98197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 --2- 172.21.15.53:0/1897761668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a980ffdb0 0x7f3a981981d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:35.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.958+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 >> 172.21.15.53:0/1897761668 conn(0x7f3a980fa1e0 msgr2=0x7f3a98068120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:35.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.959+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 shutdown_connections 2023-07-14T19:24:35.962 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:35.959+0000 7f3a9ce1d700 1 -- 172.21.15.53:0/1897761668 wait complete. 2023-07-14T19:24:36.332 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ps' 2023-07-14T19:24:36.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:36 smithi053 ceph-mon[96565]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:36.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:36 smithi070 ceph-mon[98015]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:36.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:36 smithi104 ceph-mon[98435]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:36.894 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:38.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 -- 172.21.15.53:0/1713871342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab0100740 msgr2=0x7f2ab0107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:38.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/1713871342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab0100740 0x7f2ab0107000 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0009990 tx=0x7f2aa0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 -- 172.21.15.53:0/1713871342 shutdown_connections 2023-07-14T19:24:38.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/1713871342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab0100740 0x7f2ab0107000 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/1713871342 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab00febb0 0x7f2ab00fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/1713871342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ab00ffdb0 0x7f2ab0100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 -- 172.21.15.53:0/1713871342 >> 172.21.15.53:0/1713871342 conn(0x7f2ab00fa1e0 msgr2=0x7f2ab00fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 -- 172.21.15.53:0/1713871342 shutdown_connections 2023-07-14T19:24:38.079 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 -- 172.21.15.53:0/1713871342 wait complete. 2023-07-14T19:24:38.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.074+0000 7f2ab76f4700 1 Processor -- start 2023-07-14T19:24:38.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab76f4700 1 -- start start 2023-07-14T19:24:38.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab76f4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00febb0 0x7f2ab0102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:38.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab76f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab00ffdb0 0x7f2ab0101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:38.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab76f4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ab0100740 0x7f2ab01018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:38.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab76f4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab0102030 con 0x7f2ab00febb0 2023-07-14T19:24:38.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab76f4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab01021a0 con 0x7f2ab0100740 2023-07-14T19:24:38.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab76f4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab019f8e0 con 0x7f2ab00ffdb0 2023-07-14T19:24:38.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5490700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00febb0 0x7f2ab0102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:38.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5490700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00febb0 0x7f2ab0102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51156/0 (socket says 172.21.15.53:51156) 2023-07-14T19:24:38.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5490700 1 -- 172.21.15.53:0/2804459061 learned_addr learned my addr 172.21.15.53:0/2804459061 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:38.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5c91700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ab0100740 0x7f2ab01018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:38.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab4c8f700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab00ffdb0 0x7f2ab0101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:38.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5490700 1 -- 172.21.15.53:0/2804459061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab00ffdb0 msgr2=0x7f2ab0101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:38.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5490700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab00ffdb0 0x7f2ab0101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.083 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5490700 1 -- 172.21.15.53:0/2804459061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ab0100740 msgr2=0x7f2ab01018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:38.083 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5490700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ab0100740 0x7f2ab01018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.083 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5490700 1 -- 172.21.15.53:0/2804459061 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa00095f0 con 0x7f2ab00febb0 2023-07-14T19:24:38.083 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.075+0000 7f2ab5490700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00febb0 0x7f2ab0102d20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8007e00 tx=0x7f2aa800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:38.084 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.076+0000 7f2aae7fc700 1 -- 172.21.15.53:0/2804459061 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800ad40 con 0x7f2ab00febb0 2023-07-14T19:24:38.084 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.076+0000 7f2aae7fc700 1 -- 172.21.15.53:0/2804459061 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2aa80128e0 con 0x7f2ab00febb0 2023-07-14T19:24:38.084 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.076+0000 7f2aae7fc700 1 -- 172.21.15.53:0/2804459061 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8011760 con 0x7f2ab00febb0 2023-07-14T19:24:38.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.076+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab019fbc0 con 0x7f2ab00febb0 2023-07-14T19:24:38.086 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.076+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab01a00c0 con 0x7f2ab00febb0 2023-07-14T19:24:38.087 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.077+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab0106b70 con 0x7f2ab00febb0 2023-07-14T19:24:38.087 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.078+0000 7f2aae7fc700 1 -- 172.21.15.53:0/2804459061 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2aa8012a50 con 0x7f2ab00febb0 2023-07-14T19:24:38.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.078+0000 7f2aae7fc700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2a980a1260 0x7f2a980a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:38.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.078+0000 7f2ab4c8f700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2a980a1260 0x7f2a980a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:38.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.079+0000 7f2aae7fc700 1 -- 172.21.15.53:0/2804459061 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2aa80ccb60 con 0x7f2ab00febb0 2023-07-14T19:24:38.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.079+0000 7f2ab4c8f700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2a980a1260 0x7f2a980a3710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4005950 tx=0x7f2aa40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:38.089 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.082+0000 7f2aae7fc700 1 -- 172.21.15.53:0/2804459061 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2aa8096310 con 0x7f2ab00febb0 2023-07-14T19:24:38.291 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.289+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ab0060a80 con 0x7f2a980a1260 2023-07-14T19:24:38.312 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.310+0000 7f2aae7fc700 1 -- 172.21.15.53:0/2804459061 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f2ab0060a80 con 0x7f2a980a1260 2023-07-14T19:24:38.313 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-14T19:24:38.313 INFO:teuthology.orchestra.run.smithi053.stdout:mds.a smithi053 running (68s) - 68s 14.7M - 18.0.0-4940-g81c9387d 219b180907a8 e3061997a51f 2023-07-14T19:24:38.313 INFO:teuthology.orchestra.run.smithi053.stdout:mds.b smithi070 running (66s) - 66s 15.7M - 18.0.0-4940-g81c9387d 219b180907a8 bfdb97c1e189 2023-07-14T19:24:38.313 INFO:teuthology.orchestra.run.smithi053.stdout:mds.c smithi104 running (77s) - 77s 14.3M - 18.0.0-4940-g81c9387d 219b180907a8 40a7f7a2daa3 2023-07-14T19:24:38.313 INFO:teuthology.orchestra.run.smithi053.stdout:mds.d smithi053 running (84s) - 84s 15.7M - 18.0.0-4940-g81c9387d 219b180907a8 095b13036b09 2023-07-14T19:24:38.314 INFO:teuthology.orchestra.run.smithi053.stdout:mds.e smithi070 running (82s) - 81s 13.4M - 18.0.0-4940-g81c9387d 219b180907a8 2d1155c1a53f 2023-07-14T19:24:38.314 INFO:teuthology.orchestra.run.smithi053.stdout:mds.f smithi104 running (80s) - 79s 16.5M - 18.0.0-4940-g81c9387d 219b180907a8 ddc30f5dbf73 2023-07-14T19:24:38.314 INFO:teuthology.orchestra.run.smithi053.stdout:mds.g smithi053 running (71s) - 70s 13.2M - 18.0.0-4940-g81c9387d 219b180907a8 5d6e73cf16d0 2023-07-14T19:24:38.314 INFO:teuthology.orchestra.run.smithi053.stdout:mds.h smithi070 running (75s) - 75s 15.7M - 18.0.0-4940-g81c9387d 219b180907a8 ceb92a0e25a0 2023-07-14T19:24:38.314 INFO:teuthology.orchestra.run.smithi053.stdout:mds.i smithi104 running (73s) - 73s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 7bdb0c377ab2 2023-07-14T19:24:38.314 INFO:teuthology.orchestra.run.smithi053.stdout:mds.j smithi053 running (86s) - 86s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 50f1864c3863 2023-07-14T19:24:38.315 INFO:teuthology.orchestra.run.smithi053.stdout:mds.k smithi070 running (88s) - 88s 16.7M - 18.0.0-4940-g81c9387d 219b180907a8 72d94f7f2e55 2023-07-14T19:24:38.315 INFO:teuthology.orchestra.run.smithi053.stdout:mds.l smithi104 running (90s) - 90s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 c35183171685 2023-07-14T19:24:38.315 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.x smithi053 *:9283,8765 running (7m) - 7m 511M - 18.0.0-4940-g81c9387d 219b180907a8 f89b8bbe9363 2023-07-14T19:24:38.315 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.y smithi070 *:8443,8765 running (5m) - 5m 446M - 18.0.0-4940-g81c9387d 219b180907a8 2664713cf13b 2023-07-14T19:24:38.315 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.z smithi104 *:8443,8765 running (5m) - 5m 448M - 18.0.0-4940-g81c9387d 219b180907a8 9c4785611917 2023-07-14T19:24:38.315 INFO:teuthology.orchestra.run.smithi053.stdout:mon.a smithi053 running (7m) - 7m 86.8M 2048M 18.0.0-4940-g81c9387d 219b180907a8 b0966646b52b 2023-07-14T19:24:38.315 INFO:teuthology.orchestra.run.smithi053.stdout:mon.b smithi070 running (6m) - 6m 57.6M 2048M 18.0.0-4940-g81c9387d 219b180907a8 1eface503eae 2023-07-14T19:24:38.316 INFO:teuthology.orchestra.run.smithi053.stdout:mon.c smithi104 running (6m) - 6m 60.8M 2048M 18.0.0-4940-g81c9387d 219b180907a8 6c421be06c2c 2023-07-14T19:24:38.316 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (5m) - 5m 55.0M 4293M 18.0.0-4940-g81c9387d 219b180907a8 1d5d45bf8aa7 2023-07-14T19:24:38.316 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi070 running (5m) - 5m 62.0M 4292M 18.0.0-4940-g81c9387d 219b180907a8 484f8f53308a 2023-07-14T19:24:38.316 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi104 running (4m) - 4m 54.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 c8eec84b58f6 2023-07-14T19:24:38.317 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 smithi053 running (4m) - 4m 52.8M 4293M 18.0.0-4940-g81c9387d 219b180907a8 c89fe8ebb0de 2023-07-14T19:24:38.317 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 smithi070 running (4m) - 4m 48.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 a7ecaf1cad7f 2023-07-14T19:24:38.317 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 smithi104 running (3m) - 3m 51.6M 4292M 18.0.0-4940-g81c9387d 219b180907a8 0dfe1be069ed 2023-07-14T19:24:38.317 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 smithi053 running (3m) - 3m 47.9M 4293M 18.0.0-4940-g81c9387d 219b180907a8 655d7d636049 2023-07-14T19:24:38.318 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 smithi070 running (3m) - 3m 53.1M 4292M 18.0.0-4940-g81c9387d 219b180907a8 7c2bbfd90fc7 2023-07-14T19:24:38.318 INFO:teuthology.orchestra.run.smithi053.stdout:osd.8 smithi104 running (2m) - 2m 51.7M 4292M 18.0.0-4940-g81c9387d 219b180907a8 c7124b9dbbb8 2023-07-14T19:24:38.318 INFO:teuthology.orchestra.run.smithi053.stdout:osd.9 smithi053 running (2m) - 2m 48.3M 4293M 18.0.0-4940-g81c9387d 219b180907a8 bcba98715318 2023-07-14T19:24:38.318 INFO:teuthology.orchestra.run.smithi053.stdout:osd.10 smithi070 running (2m) - 2m 50.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 16a542fde83e 2023-07-14T19:24:38.319 INFO:teuthology.orchestra.run.smithi053.stdout:osd.11 smithi104 running (109s) - 108s 44.7M 4292M 18.0.0-4940-g81c9387d 219b180907a8 09509671f9fd 2023-07-14T19:24:38.320 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2a980a1260 msgr2=0x7f2a980a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:38.320 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2a980a1260 0x7f2a980a3710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4005950 tx=0x7f2aa40058e0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.320 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00febb0 msgr2=0x7f2ab0102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:38.321 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00febb0 0x7f2ab0102d20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8007e00 tx=0x7f2aa800a340 comp rx=0 tx=0).stop 2023-07-14T19:24:38.321 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 shutdown_connections 2023-07-14T19:24:38.321 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f2a980a1260 0x7f2a980a3710 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.321 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ab0100740 0x7f2ab01018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.322 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00febb0 0x7f2ab0102d20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.322 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 --2- 172.21.15.53:0/2804459061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ab00ffdb0 0x7f2ab0101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:38.322 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 >> 172.21.15.53:0/2804459061 conn(0x7f2ab00fa1e0 msgr2=0x7f2ab00fc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:38.322 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 shutdown_connections 2023-07-14T19:24:38.322 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:38.313+0000 7f2ab76f4700 1 -- 172.21.15.53:0/2804459061 wait complete. 2023-07-14T19:24:38.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:38 smithi053 ceph-mon[96565]: pgmap v278: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:24:38.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:38 smithi070 ceph-mon[98015]: pgmap v278: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:24:38.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:38 smithi104 ceph-mon[98435]: pgmap v278: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:24:38.853 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ls' 2023-07-14T19:24:39.418 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:39.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:39 smithi053 ceph-mon[96565]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:39.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:39 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:39.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:39 smithi070 ceph-mon[98015]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:39.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:39 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:39.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:39 smithi104 ceph-mon[98435]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:39.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:39 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:40.473 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:40 smithi053 ceph-mon[96565]: pgmap v279: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:24:40.660 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.634+0000 7f94b0c97700 1 -- 172.21.15.53:0/136039400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0ffe40 msgr2=0x7f94ac100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:40.660 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.634+0000 7f94b0c97700 1 --2- 172.21.15.53:0/136039400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0ffe40 0x7f94ac100290 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9494009990 tx=0x7f9494009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.660 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.634+0000 7f94b0c97700 1 -- 172.21.15.53:0/136039400 shutdown_connections 2023-07-14T19:24:40.660 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.634+0000 7f94b0c97700 1 --2- 172.21.15.53:0/136039400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94ac1007d0 0x7f94ac107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.661 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.634+0000 7f94b0c97700 1 --2- 172.21.15.53:0/136039400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac0fec40 0x7f94ac0ff050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.661 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.634+0000 7f94b0c97700 1 --2- 172.21.15.53:0/136039400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0ffe40 0x7f94ac100290 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.661 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.634+0000 7f94b0c97700 1 -- 172.21.15.53:0/136039400 >> 172.21.15.53:0/136039400 conn(0x7f94ac0fa270 msgr2=0x7f94ac0fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:40.661 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.634+0000 7f94b0c97700 1 -- 172.21.15.53:0/136039400 shutdown_connections 2023-07-14T19:24:40.661 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.634+0000 7f94b0c97700 1 -- 172.21.15.53:0/136039400 wait complete. 2023-07-14T19:24:40.661 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.635+0000 7f94b0c97700 1 Processor -- start 2023-07-14T19:24:40.661 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.635+0000 7f94b0c97700 1 -- start start 2023-07-14T19:24:40.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.635+0000 7f94b0c97700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0fec40 0x7f94ac069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:40.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.635+0000 7f94b0c97700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac0ffe40 0x7f94ac19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:40.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.635+0000 7f94b0c97700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94ac1007d0 0x7f94ac19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:40.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.635+0000 7f94b0c97700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac19d520 con 0x7f94ac0fec40 2023-07-14T19:24:40.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.635+0000 7f94b0c97700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac19d690 con 0x7f94ac1007d0 2023-07-14T19:24:40.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.635+0000 7f94b0c97700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac19d850 con 0x7f94ac0ffe40 2023-07-14T19:24:40.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac0ffe40 0x7f94ac19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:40.662 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94aa59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0fec40 0x7f94ac069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:40.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a9d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac0ffe40 0x7f94ac19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.53:51418/0 (socket says 172.21.15.53:51418) 2023-07-14T19:24:40.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a9d9b700 1 -- 172.21.15.53:0/337172897 learned_addr learned my addr 172.21.15.53:0/337172897 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:40.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94aa59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0fec40 0x7f94ac069620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51168/0 (socket says 172.21.15.53:51168) 2023-07-14T19:24:40.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94aad9d700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94ac1007d0 0x7f94ac19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:40.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a9d9b700 1 -- 172.21.15.53:0/337172897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94ac1007d0 msgr2=0x7f94ac19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:40.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a9d9b700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94ac1007d0 0x7f94ac19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.663 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a9d9b700 1 -- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0fec40 msgr2=0x7f94ac069620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:40.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a9d9b700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0fec40 0x7f94ac069620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a9d9b700 1 -- 172.21.15.53:0/337172897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940095f0 con 0x7f94ac0ffe40 2023-07-14T19:24:40.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94aa59c700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0fec40 0x7f94ac069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:24:40.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a9d9b700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac0ffe40 0x7f94ac19a730 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9494000c00 tx=0x7f949400f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:40.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a37fe700 1 -- 172.21.15.53:0/337172897 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949401d070 con 0x7f94ac0ffe40 2023-07-14T19:24:40.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a37fe700 1 -- 172.21.15.53:0/337172897 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f949400fb50 con 0x7f94ac0ffe40 2023-07-14T19:24:40.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac19dad0 con 0x7f94ac0ffe40 2023-07-14T19:24:40.664 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94a37fe700 1 -- 172.21.15.53:0/337172897 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94940173f0 con 0x7f94ac0ffe40 2023-07-14T19:24:40.665 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.636+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac19df70 con 0x7f94ac0ffe40 2023-07-14T19:24:40.665 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.637+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac10a480 con 0x7f94ac0ffe40 2023-07-14T19:24:40.665 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.638+0000 7f94a37fe700 1 -- 172.21.15.53:0/337172897 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f949400fcc0 con 0x7f94ac0ffe40 2023-07-14T19:24:40.665 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.639+0000 7f94a37fe700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f94980a11e0 0x7f94980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:40.665 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.639+0000 7f94aa59c700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f94980a11e0 0x7f94980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:40.665 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.639+0000 7f94a37fe700 1 -- 172.21.15.53:0/337172897 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f94940cdf90 con 0x7f94ac0ffe40 2023-07-14T19:24:40.665 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.639+0000 7f94aa59c700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f94980a11e0 0x7f94980a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0ffca0 tx=0x7f949c0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:40.666 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.642+0000 7f94a37fe700 1 -- 172.21.15.53:0/337172897 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f94940977c0 con 0x7f94ac0ffe40 2023-07-14T19:24:40.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:40 smithi070 ceph-mon[98015]: pgmap v279: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:24:40.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:40 smithi104 ceph-mon[98435]: pgmap v279: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:24:40.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.872+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f94ac060a80 con 0x7f94980a11e0 2023-07-14T19:24:40.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.878+0000 7f94a37fe700 1 -- 172.21.15.53:0/337172897 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f94ac060a80 con 0x7f94980a11e0 2023-07-14T19:24:40.880 INFO:teuthology.orchestra.run.smithi053.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-07-14T19:24:40.880 INFO:teuthology.orchestra.run.smithi053.stdout:mds.all 12/12 - 95s smithi053=a;smithi053=d;smithi053=g;smithi053=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi104=c;smithi104=f;smithi104=i;smithi104=l;count:12 2023-07-14T19:24:40.880 INFO:teuthology.orchestra.run.smithi053.stdout:mgr 3/3 - 6m smithi053=x;smithi070=y;smithi104=z;count:3 2023-07-14T19:24:40.881 INFO:teuthology.orchestra.run.smithi053.stdout:mon 3/3 - 6m smithi053:172.21.15.53=a;smithi070:172.21.15.70=b;smithi104:172.21.15.104=c;count:3 2023-07-14T19:24:40.881 INFO:teuthology.orchestra.run.smithi053.stdout:osd 12 - - 2023-07-14T19:24:40.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f94980a11e0 msgr2=0x7f94980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:40.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f94980a11e0 0x7f94980a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0ffca0 tx=0x7f949c0091b0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac0ffe40 msgr2=0x7f94ac19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:40.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac0ffe40 0x7f94ac19a730 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9494000c00 tx=0x7f949400f710 comp rx=0 tx=0).stop 2023-07-14T19:24:40.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 shutdown_connections 2023-07-14T19:24:40.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f94980a11e0 0x7f94980a3690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94ac1007d0 0x7f94ac19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94ac0fec40 0x7f94ac069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 --2- 172.21.15.53:0/337172897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ac0ffe40 0x7f94ac19a730 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:40.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 >> 172.21.15.53:0/337172897 conn(0x7f94ac0fa270 msgr2=0x7f94ac0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:40.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 shutdown_connections 2023-07-14T19:24:40.884 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:40.881+0000 7f94b0c97700 1 -- 172.21.15.53:0/337172897 wait complete. 2023-07-14T19:24:41.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- bash -c 'ceph orch host ls' 2023-07-14T19:24:41.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2023-07-14T19:24:41.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2023-07-14T19:24:41.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2023-07-14T19:24:41.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2023-07-14T19:24:41.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-07-14T19:24:41.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2023-07-14T19:24:41.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:41 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2023-07-14T19:24:41.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2023-07-14T19:24:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2023-07-14T19:24:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2023-07-14T19:24:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2023-07-14T19:24:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-07-14T19:24:41.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2023-07-14T19:24:41.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:41 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2023-07-14T19:24:41.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2023-07-14T19:24:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2023-07-14T19:24:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2023-07-14T19:24:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2023-07-14T19:24:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2023-07-14T19:24:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2023-07-14T19:24:41.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:41 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2023-07-14T19:24:41.904 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:42.346 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:42.346 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2023-07-14T19:24:42.346 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2023-07-14T19:24:42.347 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2023-07-14T19:24:42.347 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2023-07-14T19:24:42.347 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-07-14T19:24:42.347 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2023-07-14T19:24:42.347 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2023-07-14T19:24:42.347 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:24:42.347 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: pgmap v281: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-07-14T19:24:42.348 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.348 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.348 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.348 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:42 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:42.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2023-07-14T19:24:42.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2023-07-14T19:24:42.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2023-07-14T19:24:42.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2023-07-14T19:24:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-07-14T19:24:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2023-07-14T19:24:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2023-07-14T19:24:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:24:42.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: pgmap v281: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-07-14T19:24:42.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:42 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:42.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2023-07-14T19:24:42.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2023-07-14T19:24:42.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2023-07-14T19:24:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2023-07-14T19:24:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2023-07-14T19:24:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2023-07-14T19:24:42.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2023-07-14T19:24:42.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:24:42.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: pgmap v281: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-07-14T19:24:42.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:42.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:42 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:43.131 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.129+0000 7f33b47c5700 1 -- 172.21.15.53:0/817721542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 msgr2=0x7f33ac107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.129+0000 7f33b47c5700 1 --2- 172.21.15.53:0/817721542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 0x7f33ac107000 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f33a400b0a0 tx=0x7f33a400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.129+0000 7f33b47c5700 1 -- 172.21.15.53:0/817721542 shutdown_connections 2023-07-14T19:24:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.129+0000 7f33b47c5700 1 --2- 172.21.15.53:0/817721542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 0x7f33ac107000 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.129+0000 7f33b47c5700 1 --2- 172.21.15.53:0/817721542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33ac0febb0 0x7f33ac0fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.129+0000 7f33b47c5700 1 --2- 172.21.15.53:0/817721542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33ac0ffdb0 0x7f33ac100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.129+0000 7f33b47c5700 1 -- 172.21.15.53:0/817721542 >> 172.21.15.53:0/817721542 conn(0x7f33ac0fa1e0 msgr2=0x7f33ac0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.129+0000 7f33b47c5700 1 -- 172.21.15.53:0/817721542 shutdown_connections 2023-07-14T19:24:43.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.129+0000 7f33b47c5700 1 -- 172.21.15.53:0/817721542 wait complete. 2023-07-14T19:24:43.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b47c5700 1 Processor -- start 2023-07-14T19:24:43.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b47c5700 1 -- start start 2023-07-14T19:24:43.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b47c5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33ac0febb0 0x7f33ac068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:43.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b47c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33ac0ffdb0 0x7f33ac19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:43.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b47c5700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 0x7f33ac19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:43.133 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b47c5700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ac19d600 con 0x7f33ac100740 2023-07-14T19:24:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b47c5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ac19d770 con 0x7f33ac0febb0 2023-07-14T19:24:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b47c5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ac19d930 con 0x7f33ac0ffdb0 2023-07-14T19:24:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b2d62700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 0x7f33ac19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b2561700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33ac0febb0 0x7f33ac068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b2d62700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 0x7f33ac19ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51182/0 (socket says 172.21.15.53:51182) 2023-07-14T19:24:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b2d62700 1 -- 172.21.15.53:0/4073928637 learned_addr learned my addr 172.21.15.53:0/4073928637 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b1d60700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33ac0ffdb0 0x7f33ac19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:43.135 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b2d62700 1 -- 172.21.15.53:0/4073928637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33ac0ffdb0 msgr2=0x7f33ac19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:43.136 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b2d62700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33ac0ffdb0 0x7f33ac19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.136 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b2d62700 1 -- 172.21.15.53:0/4073928637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33ac0febb0 msgr2=0x7f33ac068c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:43.136 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b2d62700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33ac0febb0 0x7f33ac068c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.136 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b2d62700 1 -- 172.21.15.53:0/4073928637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a4009d00 con 0x7f33ac100740 2023-07-14T19:24:43.136 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.130+0000 7f33b1d60700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33ac0ffdb0 0x7f33ac19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:43.136 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.131+0000 7f33b2d62700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 0x7f33ac19ce60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f33a400b780 tx=0x7f33a400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:43.137 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.131+0000 7f339f7fe700 1 -- 172.21.15.53:0/4073928637 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a400e040 con 0x7f33ac100740 2023-07-14T19:24:43.137 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.131+0000 7f339f7fe700 1 -- 172.21.15.53:0/4073928637 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33a40094a0 con 0x7f33ac100740 2023-07-14T19:24:43.137 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.131+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ac19dbb0 con 0x7f33ac100740 2023-07-14T19:24:43.137 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.131+0000 7f339f7fe700 1 -- 172.21.15.53:0/4073928637 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4005330 con 0x7f33ac100740 2023-07-14T19:24:43.137 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.131+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ac19dfd0 con 0x7f33ac100740 2023-07-14T19:24:43.137 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.132+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ac106100 con 0x7f33ac100740 2023-07-14T19:24:43.137 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.133+0000 7f339f7fe700 1 -- 172.21.15.53:0/4073928637 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f33a401da30 con 0x7f33ac100740 2023-07-14T19:24:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.133+0000 7f339f7fe700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33980a11e0 0x7f33980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.134+0000 7f33b2561700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33980a11e0 0x7f33980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.134+0000 7f339f7fe700 1 -- 172.21.15.53:0/4073928637 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f33a40ce650 con 0x7f33ac100740 2023-07-14T19:24:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.134+0000 7f33b2561700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33980a11e0 0x7f33980a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0ffc10 tx=0x7f33a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.137+0000 7f339f7fe700 1 -- 172.21.15.53:0/4073928637 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f33a4097d00 con 0x7f33ac100740 2023-07-14T19:24:43.348 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.347+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f33ac060a80 con 0x7f33980a11e0 2023-07-14T19:24:43.350 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.349+0000 7f339f7fe700 1 -- 172.21.15.53:0/4073928637 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f33ac060a80 con 0x7f33980a11e0 2023-07-14T19:24:43.351 INFO:teuthology.orchestra.run.smithi053.stdout:HOST ADDR LABELS STATUS 2023-07-14T19:24:43.351 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 172.21.15.53 2023-07-14T19:24:43.351 INFO:teuthology.orchestra.run.smithi053.stdout:smithi070 172.21.15.70 2023-07-14T19:24:43.351 INFO:teuthology.orchestra.run.smithi053.stdout:smithi104 172.21.15.104 2023-07-14T19:24:43.351 INFO:teuthology.orchestra.run.smithi053.stdout:3 hosts in cluster 2023-07-14T19:24:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33980a11e0 msgr2=0x7f33980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33980a11e0 0x7f33980a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0ffc10 tx=0x7f33a0009260 comp rx=0 tx=0).stop 2023-07-14T19:24:43.353 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 msgr2=0x7f33ac19ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:43.353 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 0x7f33ac19ce60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f33a400b780 tx=0x7f33a400bd60 comp rx=0 tx=0).stop 2023-07-14T19:24:43.353 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 shutdown_connections 2023-07-14T19:24:43.353 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f33980a11e0 0x7f33980a3690 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.353 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33ac100740 0x7f33ac19ce60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33ac0febb0 0x7f33ac068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 --2- 172.21.15.53:0/4073928637 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33ac0ffdb0 0x7f33ac19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 >> 172.21.15.53:0/4073928637 conn(0x7f33ac0fa1e0 msgr2=0x7f33ac103240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 shutdown_connections 2023-07-14T19:24:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:43.351+0000 7f33b47c5700 1 -- 172.21.15.53:0/4073928637 wait complete. 2023-07-14T19:24:43.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:43 smithi053 ceph-mon[96565]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:24:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:43 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:43 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:43 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:43 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:43.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:43 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:43.653 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- bash -c 'ceph orch device ls' 2023-07-14T19:24:43.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:43 smithi070 ceph-mon[98015]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:24:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:43 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:43 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:43 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:43 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:43.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:43 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:43.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:43 smithi104 ceph-mon[98435]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:24:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:43 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:43 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:43 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:43 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:43.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:43 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' 2023-07-14T19:24:44.214 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:44.685 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:44 smithi053 ceph-mon[96565]: pgmap v283: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2023-07-14T19:24:44.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:44 smithi070 ceph-mon[98015]: pgmap v283: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2023-07-14T19:24:44.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:44 smithi104 ceph-mon[98435]: pgmap v283: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2023-07-14T19:24:45.397 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.392+0000 7f9057a6a700 1 -- 172.21.15.53:0/1823974465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9050100740 msgr2=0x7f9050107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.392+0000 7f9057a6a700 1 --2- 172.21.15.53:0/1823974465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9050100740 0x7f9050107000 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f904800b0a0 tx=0x7f904800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.398 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.392+0000 7f9057a6a700 1 -- 172.21.15.53:0/1823974465 shutdown_connections 2023-07-14T19:24:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.392+0000 7f9057a6a700 1 --2- 172.21.15.53:0/1823974465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9050100740 0x7f9050107000 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.393+0000 7f9057a6a700 1 --2- 172.21.15.53:0/1823974465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90500febb0 0x7f90500fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.393+0000 7f9057a6a700 1 --2- 172.21.15.53:0/1823974465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90500ffdb0 0x7f9050100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.393+0000 7f9057a6a700 1 -- 172.21.15.53:0/1823974465 >> 172.21.15.53:0/1823974465 conn(0x7f90500fa1e0 msgr2=0x7f90500fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.393+0000 7f9057a6a700 1 -- 172.21.15.53:0/1823974465 shutdown_connections 2023-07-14T19:24:45.399 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.393+0000 7f9057a6a700 1 -- 172.21.15.53:0/1823974465 wait complete. 2023-07-14T19:24:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.393+0000 7f9057a6a700 1 Processor -- start 2023-07-14T19:24:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9057a6a700 1 -- start start 2023-07-14T19:24:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9057a6a700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90500febb0 0x7f9050194990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9057a6a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90500ffdb0 0x7f9050194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9057a6a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9050100740 0x7f905019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:45.400 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9057a6a700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905019c520 con 0x7f90500febb0 2023-07-14T19:24:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9057a6a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905019c660 con 0x7f9050100740 2023-07-14T19:24:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9057a6a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9050068100 con 0x7f90500ffdb0 2023-07-14T19:24:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055806700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90500febb0 0x7f9050194990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055806700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90500febb0 0x7f9050194990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51188/0 (socket says 172.21.15.53:51188) 2023-07-14T19:24:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055806700 1 -- 172.21.15.53:0/75210018 learned_addr learned my addr 172.21.15.53:0/75210018 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9056007700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9050100740 0x7f905019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:45.401 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055005700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90500ffdb0 0x7f9050194ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055806700 1 -- 172.21.15.53:0/75210018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90500ffdb0 msgr2=0x7f9050194ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055806700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90500ffdb0 0x7f9050194ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055806700 1 -- 172.21.15.53:0/75210018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9050100740 msgr2=0x7f905019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055806700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9050100740 0x7f905019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055806700 1 -- 172.21.15.53:0/75210018 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9048009d00 con 0x7f90500febb0 2023-07-14T19:24:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9056007700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9050100740 0x7f905019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:45.402 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.394+0000 7f9055806700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90500febb0 0x7f9050194990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e9a0 tx=0x7f904c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.395+0000 7f9046ffd700 1 -- 172.21.15.53:0/75210018 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c009790 con 0x7f90500febb0 2023-07-14T19:24:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.395+0000 7f9046ffd700 1 -- 172.21.15.53:0/75210018 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f904c0044e0 con 0x7f90500febb0 2023-07-14T19:24:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.395+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90500683e0 con 0x7f90500febb0 2023-07-14T19:24:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.395+0000 7f9046ffd700 1 -- 172.21.15.53:0/75210018 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c003d70 con 0x7f90500febb0 2023-07-14T19:24:45.403 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.395+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9050068930 con 0x7f90500febb0 2023-07-14T19:24:45.404 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.396+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9050106130 con 0x7f90500febb0 2023-07-14T19:24:45.405 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.397+0000 7f9046ffd700 1 -- 172.21.15.53:0/75210018 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f904c004020 con 0x7f90500febb0 2023-07-14T19:24:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.398+0000 7f9046ffd700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f903c0a1260 0x7f903c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.398+0000 7f9055005700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f903c0a1260 0x7f903c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.398+0000 7f9046ffd700 1 -- 172.21.15.53:0/75210018 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f904c014070 con 0x7f90500febb0 2023-07-14T19:24:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.398+0000 7f9055005700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f903c0a1260 0x7f903c0a3710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9040009af0 tx=0x7f90400094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:45.406 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.400+0000 7f9046ffd700 1 -- 172.21.15.53:0/75210018 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f904c050780 con 0x7f90500febb0 2023-07-14T19:24:45.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:45 smithi053 ceph-mon[96565]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:45.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:45 smithi053 ceph-mon[96565]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-07-14T19:24:45.605 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.604+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f905002c9f0 con 0x7f903c0a1260 2023-07-14T19:24:45.607 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.606+0000 7f9046ffd700 1 -- 172.21.15.53:0/75210018 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f905002c9f0 con 0x7f903c0a1260 2023-07-14T19:24:45.607 INFO:teuthology.orchestra.run.smithi053.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-07-14T19:24:45.608 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300G8480DGN 447G No 2m ago LVM detected, locked 2023-07-14T19:24:45.608 INFO:teuthology.orchestra.run.smithi053.stdout:smithi070 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002P480DGN 447G No 2m ago LVM detected, locked 2023-07-14T19:24:45.608 INFO:teuthology.orchestra.run.smithi053.stdout:smithi104 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NW400BGN 372G No 110s ago LVM detected, locked 2023-07-14T19:24:45.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.608+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f903c0a1260 msgr2=0x7f903c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:45.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.608+0000 7f9057a6a700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f903c0a1260 0x7f903c0a3710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9040009af0 tx=0x7f90400094f0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.608+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90500febb0 msgr2=0x7f9050194990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:45.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.608+0000 7f9057a6a700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90500febb0 0x7f9050194990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e9a0 tx=0x7f904c00ed60 comp rx=0 tx=0).stop 2023-07-14T19:24:45.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.609+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 shutdown_connections 2023-07-14T19:24:45.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.609+0000 7f9057a6a700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f903c0a1260 0x7f903c0a3710 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.609+0000 7f9057a6a700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9050100740 0x7f905019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.609+0000 7f9057a6a700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90500febb0 0x7f9050194990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.609+0000 7f9057a6a700 1 --2- 172.21.15.53:0/75210018 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90500ffdb0 0x7f9050194ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:45.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.609+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 >> 172.21.15.53:0/75210018 conn(0x7f90500fa1e0 msgr2=0x7f9050103270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:45.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.609+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 shutdown_connections 2023-07-14T19:24:45.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:45.609+0000 7f9057a6a700 1 -- 172.21.15.53:0/75210018 wait complete. 2023-07-14T19:24:45.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:45 smithi070 ceph-mon[98015]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:45.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:45 smithi070 ceph-mon[98015]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-07-14T19:24:45.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:45 smithi104 ceph-mon[98435]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:45.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:45 smithi104 ceph-mon[98435]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-07-14T19:24:45.974 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T19:24:45.986 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi053.front.sepia.ceph.com 2023-07-14T19:24:45.986 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- bash -c 'ceph fs dump' 2023-07-14T19:24:46.546 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:46.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:46 smithi070 ceph-mon[98015]: pgmap v284: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T19:24:46.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:46 smithi104 ceph-mon[98435]: pgmap v284: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T19:24:46.808 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:46 smithi053 ceph-mon[96565]: pgmap v284: 97 pgs: 1 peering, 96 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T19:24:47.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:47 smithi053 ceph-mon[96565]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:47.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:47 smithi070 ceph-mon[98015]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:47.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:47 smithi104 ceph-mon[98435]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:47.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.807+0000 7f838a670700 1 -- 172.21.15.53:0/1795521613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 msgr2=0x7f8384100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:47.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.807+0000 7f838a670700 1 --2- 172.21.15.53:0/1795521613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 0x7f8384100200 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8378009990 tx=0x7f8378009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:47.810 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.807+0000 7f838a670700 1 -- 172.21.15.53:0/1795521613 shutdown_connections 2023-07-14T19:24:47.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.807+0000 7f838a670700 1 --2- 172.21.15.53:0/1795521613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8384100740 0x7f8384107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:47.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.807+0000 7f838a670700 1 --2- 172.21.15.53:0/1795521613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83840febb0 0x7f83840fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:47.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.807+0000 7f838a670700 1 --2- 172.21.15.53:0/1795521613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 0x7f8384100200 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:47.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.807+0000 7f838a670700 1 -- 172.21.15.53:0/1795521613 >> 172.21.15.53:0/1795521613 conn(0x7f83840fa1e0 msgr2=0x7f83840fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:47.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.807+0000 7f838a670700 1 -- 172.21.15.53:0/1795521613 shutdown_connections 2023-07-14T19:24:47.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.807+0000 7f838a670700 1 -- 172.21.15.53:0/1795521613 wait complete. 2023-07-14T19:24:47.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.808+0000 7f838a670700 1 Processor -- start 2023-07-14T19:24:47.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.808+0000 7f838a670700 1 -- start start 2023-07-14T19:24:47.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.808+0000 7f838a670700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83840febb0 0x7f8384197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:47.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.808+0000 7f838a670700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 0x7f83841980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:47.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.808+0000 7f838a670700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8384100740 0x7f838419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:47.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.808+0000 7f838a670700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838419f720 con 0x7f83840ffdb0 2023-07-14T19:24:47.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.808+0000 7f838a670700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838419f860 con 0x7f83840febb0 2023-07-14T19:24:47.812 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.808+0000 7f838a670700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838419f9f0 con 0x7f8384100740 2023-07-14T19:24:47.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83837fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 0x7f83841980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:47.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83837fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 0x7f83841980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51204/0 (socket says 172.21.15.53:51204) 2023-07-14T19:24:47.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83837fe700 1 -- 172.21.15.53:0/736495716 learned_addr learned my addr 172.21.15.53:0/736495716 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:47.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f8388c0d700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8384100740 0x7f838419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:47.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f8383fff700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83840febb0 0x7f8384197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:47.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83837fe700 1 -- 172.21.15.53:0/736495716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8384100740 msgr2=0x7f838419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:47.813 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83837fe700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8384100740 0x7f838419f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:47.814 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83837fe700 1 -- 172.21.15.53:0/736495716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83840febb0 msgr2=0x7f8384197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:47.814 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83837fe700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83840febb0 0x7f8384197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:47.814 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83837fe700 1 -- 172.21.15.53:0/736495716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83780095f0 con 0x7f83840ffdb0 2023-07-14T19:24:47.814 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83837fe700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 0x7f83841980d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8378005910 tx=0x7f837800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:47.815 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83817fa700 1 -- 172.21.15.53:0/736495716 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837801d070 con 0x7f83840ffdb0 2023-07-14T19:24:47.815 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83817fa700 1 -- 172.21.15.53:0/736495716 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8378022470 con 0x7f83840ffdb0 2023-07-14T19:24:47.816 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838419fc70 con 0x7f83840ffdb0 2023-07-14T19:24:47.816 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f83817fa700 1 -- 172.21.15.53:0/736495716 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378017650 con 0x7f83840ffdb0 2023-07-14T19:24:47.816 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.809+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83841a0160 con 0x7f83840ffdb0 2023-07-14T19:24:47.816 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.810+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838410a370 con 0x7f83840ffdb0 2023-07-14T19:24:47.816 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.811+0000 7f83817fa700 1 -- 172.21.15.53:0/736495716 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f83780225e0 con 0x7f83840ffdb0 2023-07-14T19:24:47.816 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.812+0000 7f83817fa700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f836c0a1260 0x7f836c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:47.816 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.812+0000 7f8383fff700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f836c0a1260 0x7f836c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:47.817 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.812+0000 7f83817fa700 1 -- 172.21.15.53:0/736495716 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f83780cdb90 con 0x7f83840ffdb0 2023-07-14T19:24:47.817 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.812+0000 7f8383fff700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f836c0a1260 0x7f836c0a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f83840ffc10 tx=0x7f837400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:47.817 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:47.816+0000 7f83817fa700 1 -- 172.21.15.53:0/736495716 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8378097360 con 0x7f83840ffdb0 2023-07-14T19:24:48.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.057+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f83841a0710 con 0x7f83840ffdb0 2023-07-14T19:24:48.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.057+0000 7f83817fa700 1 -- 172.21.15.53:0/736495716 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+2893 (secure 0 0 0) 0x7f8378027070 con 0x7f83840ffdb0 2023-07-14T19:24:48.059 INFO:teuthology.orchestra.run.smithi053.stdout:e23 2023-07-14T19:24:48.059 INFO:teuthology.orchestra.run.smithi053.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:24:48.059 INFO:teuthology.orchestra.run.smithi053.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:24:48.060 INFO:teuthology.orchestra.run.smithi053.stdout:legacy client fscid: 1 2023-07-14T19:24:48.060 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:48.060 INFO:teuthology.orchestra.run.smithi053.stdout:Filesystem 'cephfs' (1) 2023-07-14T19:24:48.060 INFO:teuthology.orchestra.run.smithi053.stdout:fs_name cephfs 2023-07-14T19:24:48.060 INFO:teuthology.orchestra.run.smithi053.stdout:epoch 23 2023-07-14T19:24:48.060 INFO:teuthology.orchestra.run.smithi053.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-07-14T19:24:48.061 INFO:teuthology.orchestra.run.smithi053.stdout:created 2023-07-14T19:23:37.894565+0000 2023-07-14T19:24:48.061 INFO:teuthology.orchestra.run.smithi053.stdout:modified 2023-07-14T19:23:43.289341+0000 2023-07-14T19:24:48.061 INFO:teuthology.orchestra.run.smithi053.stdout:tableserver 0 2023-07-14T19:24:48.061 INFO:teuthology.orchestra.run.smithi053.stdout:root 0 2023-07-14T19:24:48.061 INFO:teuthology.orchestra.run.smithi053.stdout:session_timeout 300 2023-07-14T19:24:48.062 INFO:teuthology.orchestra.run.smithi053.stdout:session_autoclose 300 2023-07-14T19:24:48.062 INFO:teuthology.orchestra.run.smithi053.stdout:max_file_size 1099511627776 2023-07-14T19:24:48.062 INFO:teuthology.orchestra.run.smithi053.stdout:max_xattr_size 65536 2023-07-14T19:24:48.062 INFO:teuthology.orchestra.run.smithi053.stdout:required_client_features {} 2023-07-14T19:24:48.062 INFO:teuthology.orchestra.run.smithi053.stdout:last_failure 0 2023-07-14T19:24:48.062 INFO:teuthology.orchestra.run.smithi053.stdout:last_failure_osd_epoch 0 2023-07-14T19:24:48.063 INFO:teuthology.orchestra.run.smithi053.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:24:48.063 INFO:teuthology.orchestra.run.smithi053.stdout:max_mds 3 2023-07-14T19:24:48.063 INFO:teuthology.orchestra.run.smithi053.stdout:in 0,1,2 2023-07-14T19:24:48.063 INFO:teuthology.orchestra.run.smithi053.stdout:up {0=24527,1=24491,2=24461} 2023-07-14T19:24:48.063 INFO:teuthology.orchestra.run.smithi053.stdout:failed 2023-07-14T19:24:48.063 INFO:teuthology.orchestra.run.smithi053.stdout:damaged 2023-07-14T19:24:48.063 INFO:teuthology.orchestra.run.smithi053.stdout:stopped 2023-07-14T19:24:48.064 INFO:teuthology.orchestra.run.smithi053.stdout:data_pools [3] 2023-07-14T19:24:48.064 INFO:teuthology.orchestra.run.smithi053.stdout:metadata_pool 2 2023-07-14T19:24:48.064 INFO:teuthology.orchestra.run.smithi053.stdout:inline_data disabled 2023-07-14T19:24:48.064 INFO:teuthology.orchestra.run.smithi053.stdout:balancer 2023-07-14T19:24:48.064 INFO:teuthology.orchestra.run.smithi053.stdout:bal_rank_mask -1 2023-07-14T19:24:48.064 INFO:teuthology.orchestra.run.smithi053.stdout:standby_count_wanted 0 2023-07-14T19:24:48.064 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b{0:24527} state up:active seq 3 addr [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.065 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.i{0:24484} state up:standby-replay seq 1 addr [v2:172.21.15.104:6838/1650908884,v1:172.21.15.104:6839/1650908884] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.065 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.h{1:24491} state up:active seq 6 addr [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.065 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c{1:24463} state up:standby-replay seq 1 addr [v2:172.21.15.104:6836/1255142015,v1:172.21.15.104:6837/1255142015] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.065 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.e{2:24461} state up:active seq 8 addr [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.065 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.f{2:24451} state up:standby-replay seq 1 addr [v2:172.21.15.104:6834/1013468175,v1:172.21.15.104:6835/1013468175] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.065 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:48.065 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:48.065 INFO:teuthology.orchestra.run.smithi053.stdout:Standby daemons: 2023-07-14T19:24:48.066 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:48.066 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.j{-1:14559} state up:standby seq 1 addr [v2:172.21.15.53:6834/2670019040,v1:172.21.15.53:6835/2670019040] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.066 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d{-1:14565} state up:standby seq 1 addr [v2:172.21.15.53:6836/1144651849,v1:172.21.15.53:6837/1144651849] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.066 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.g{-1:14622} state up:standby seq 1 addr [v2:172.21.15.53:6838/122790216,v1:172.21.15.53:6839/122790216] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.066 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a{-1:14634} state up:standby seq 1 addr [v2:172.21.15.53:6840/2523933697,v1:172.21.15.53:6841/2523933697] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.066 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.l{-1:24418} state up:standby seq 1 addr [v2:172.21.15.104:6832/947644014,v1:172.21.15.104:6833/947644014] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.066 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.k{-1:24440} state up:standby seq 1 addr [v2:172.21.15.70:6832/3920144890,v1:172.21.15.70:6833/3920144890] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:24:48.067 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.059+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f836c0a1260 msgr2=0x7f836c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:48.067 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.059+0000 7f838a670700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f836c0a1260 0x7f836c0a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f83840ffc10 tx=0x7f837400b410 comp rx=0 tx=0).stop 2023-07-14T19:24:48.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.059+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 msgr2=0x7f83841980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:48.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.059+0000 7f838a670700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 0x7f83841980d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8378005910 tx=0x7f837800f710 comp rx=0 tx=0).stop 2023-07-14T19:24:48.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.060+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 shutdown_connections 2023-07-14T19:24:48.068 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.060+0000 7f838a670700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f836c0a1260 0x7f836c0a3710 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.060+0000 7f838a670700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8384100740 0x7f838419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.060+0000 7f838a670700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83840febb0 0x7f8384197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.060+0000 7f838a670700 1 --2- 172.21.15.53:0/736495716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83840ffdb0 0x7f83841980d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.060+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 >> 172.21.15.53:0/736495716 conn(0x7f83840fa1e0 msgr2=0x7f83840faa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:48.069 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.060+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 shutdown_connections 2023-07-14T19:24:48.070 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:48.060+0000 7f838a670700 1 -- 172.21.15.53:0/736495716 wait complete. 2023-07-14T19:24:48.070 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 23 2023-07-14T19:24:48.396 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:48 smithi053 ceph-mon[96565]: pgmap v285: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 86 B/s, 1 objects/s recovering 2023-07-14T19:24:48.397 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:48 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/736495716' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:24:48.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c5147628-227a-11ee-9b33-001a4aab830c -- bash -c 'ceph osd dump' 2023-07-14T19:24:48.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:48 smithi070 ceph-mon[98015]: pgmap v285: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 86 B/s, 1 objects/s recovering 2023-07-14T19:24:48.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:48 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/736495716' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:24:48.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:48 smithi104 ceph-mon[98435]: pgmap v285: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 86 B/s, 1 objects/s recovering 2023-07-14T19:24:48.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:48 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/736495716' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:24:48.970 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:49.469 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:49 smithi053 ceph-mon[96565]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-07-14T19:24:49.469 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:49 smithi053 ceph-mon[96565]: Cluster is now healthy 2023-07-14T19:24:49.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:49 smithi070 ceph-mon[98015]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-07-14T19:24:49.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:49 smithi070 ceph-mon[98015]: Cluster is now healthy 2023-07-14T19:24:49.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:49 smithi104 ceph-mon[98435]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-07-14T19:24:49.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:49 smithi104 ceph-mon[98435]: Cluster is now healthy 2023-07-14T19:24:50.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.169+0000 7f69a8492700 1 -- 172.21.15.53:0/1026348485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00fe5b0 msgr2=0x7f69a00fe9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:50.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.169+0000 7f69a8492700 1 --2- 172.21.15.53:0/1026348485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00fe5b0 0x7f69a00fe9c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6994009990 tx=0x7f6994009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.173 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.169+0000 7f69a8492700 1 -- 172.21.15.53:0/1026348485 shutdown_connections 2023-07-14T19:24:50.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.169+0000 7f69a8492700 1 --2- 172.21.15.53:0/1026348485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a00ff8b0 0x7f69a00ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.169+0000 7f69a8492700 1 --2- 172.21.15.53:0/1026348485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00fe5b0 0x7f69a00fe9c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.169+0000 7f69a8492700 1 --2- 172.21.15.53:0/1026348485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69a00fef00 0x7f69a00ff370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.169+0000 7f69a8492700 1 -- 172.21.15.53:0/1026348485 >> 172.21.15.53:0/1026348485 conn(0x7f69a00fa1e0 msgr2=0x7f69a00fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:50.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.169+0000 7f69a8492700 1 -- 172.21.15.53:0/1026348485 shutdown_connections 2023-07-14T19:24:50.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.169+0000 7f69a8492700 1 -- 172.21.15.53:0/1026348485 wait complete. 2023-07-14T19:24:50.174 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a8492700 1 Processor -- start 2023-07-14T19:24:50.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a8492700 1 -- start start 2023-07-14T19:24:50.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a8492700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69a00fe5b0 0x7f69a0197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:50.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a8492700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a00fef00 0x7f69a01982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:50.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a8492700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00ff8b0 0x7f69a019f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:50.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a8492700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a019f900 con 0x7f69a00ff8b0 2023-07-14T19:24:50.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a8492700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a019fa40 con 0x7f69a00fef00 2023-07-14T19:24:50.175 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a8492700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a019fc00 con 0x7f69a00fe5b0 2023-07-14T19:24:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a6a2f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00ff8b0 0x7f69a019f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a6a2f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00ff8b0 0x7f69a019f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51220/0 (socket says 172.21.15.53:51220) 2023-07-14T19:24:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a6a2f700 1 -- 172.21.15.53:0/2393015588 learned_addr learned my addr 172.21.15.53:0/2393015588 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a5a2d700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a00fef00 0x7f69a01982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a622e700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69a00fe5b0 0x7f69a0197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a6a2f700 1 -- 172.21.15.53:0/2393015588 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69a00fe5b0 msgr2=0x7f69a0197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a6a2f700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69a00fe5b0 0x7f69a0197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a6a2f700 1 -- 172.21.15.53:0/2393015588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a00fef00 msgr2=0x7f69a01982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a6a2f700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a00fef00 0x7f69a01982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a6a2f700 1 -- 172.21.15.53:0/2393015588 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69940095f0 con 0x7f69a00ff8b0 2023-07-14T19:24:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a5a2d700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a00fef00 0x7f69a01982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.170+0000 7f69a622e700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69a00fe5b0 0x7f69a0197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.171+0000 7f69a6a2f700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00ff8b0 0x7f69a019f330 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6998007ee0 tx=0x7f699800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.171+0000 7f69937fe700 1 -- 172.21.15.53:0/2393015588 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998017070 con 0x7f69a00ff8b0 2023-07-14T19:24:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.171+0000 7f69937fe700 1 -- 172.21.15.53:0/2393015588 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f699800f040 con 0x7f69a00ff8b0 2023-07-14T19:24:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.171+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a019fee0 con 0x7f69a00ff8b0 2023-07-14T19:24:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.171+0000 7f69937fe700 1 -- 172.21.15.53:0/2393015588 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998005740 con 0x7f69a00ff8b0 2023-07-14T19:24:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.171+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a01a02f0 con 0x7f69a00ff8b0 2023-07-14T19:24:50.179 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.172+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a0109b30 con 0x7f69a00ff8b0 2023-07-14T19:24:50.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.173+0000 7f69937fe700 1 -- 172.21.15.53:0/2393015588 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f69980058a0 con 0x7f69a00ff8b0 2023-07-14T19:24:50.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.174+0000 7f69937fe700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f698c0a1260 0x7f698c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:50.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.174+0000 7f69a622e700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f698c0a1260 0x7f698c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:50.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.174+0000 7f69937fe700 1 -- 172.21.15.53:0/2393015588 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f69980cc280 con 0x7f69a00ff8b0 2023-07-14T19:24:50.180 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.174+0000 7f69a622e700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f698c0a1260 0x7f698c0a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f699400b5c0 tx=0x7f6994005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:50.181 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.178+0000 7f69937fe700 1 -- 172.21.15.53:0/2393015588 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f69980958a0 con 0x7f69a00ff8b0 2023-07-14T19:24:50.384 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.382+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f69a002c8e0 con 0x7f69a00ff8b0 2023-07-14T19:24:50.384 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.383+0000 7f69937fe700 1 -- 172.21.15.53:0/2393015588 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v82) v1 ==== 56+0+5236 (secure 0 0 0) 0x7f6998095030 con 0x7f69a00ff8b0 2023-07-14T19:24:50.385 INFO:teuthology.orchestra.run.smithi053.stdout:epoch 82 2023-07-14T19:24:50.385 INFO:teuthology.orchestra.run.smithi053.stdout:fsid c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:24:50.385 INFO:teuthology.orchestra.run.smithi053.stdout:created 2023-07-14T19:16:47.794884+0000 2023-07-14T19:24:50.386 INFO:teuthology.orchestra.run.smithi053.stdout:modified 2023-07-14T19:24:42.310475+0000 2023-07-14T19:24:50.386 INFO:teuthology.orchestra.run.smithi053.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-07-14T19:24:50.386 INFO:teuthology.orchestra.run.smithi053.stdout:crush_version 26 2023-07-14T19:24:50.387 INFO:teuthology.orchestra.run.smithi053.stdout:full_ratio 0.95 2023-07-14T19:24:50.387 INFO:teuthology.orchestra.run.smithi053.stdout:backfillfull_ratio 0.9 2023-07-14T19:24:50.389 INFO:teuthology.orchestra.run.smithi053.stdout:nearfull_ratio 0.85 2023-07-14T19:24:50.389 INFO:teuthology.orchestra.run.smithi053.stdout:require_min_compat_client luminous 2023-07-14T19:24:50.390 INFO:teuthology.orchestra.run.smithi053.stdout:min_compat_client luminous 2023-07-14T19:24:50.390 INFO:teuthology.orchestra.run.smithi053.stdout:require_osd_release reef 2023-07-14T19:24:50.390 INFO:teuthology.orchestra.run.smithi053.stdout:stretch_mode_enabled false 2023-07-14T19:24:50.390 INFO:teuthology.orchestra.run.smithi053.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-07-14T19:24:50.390 INFO:teuthology.orchestra.run.smithi053.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-07-14T19:24:50.390 INFO:teuthology.orchestra.run.smithi053.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2023-07-14T19:24:50.390 INFO:teuthology.orchestra.run.smithi053.stdout:max_osd 12 2023-07-14T19:24:50.390 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6802/243717916,v1:172.21.15.53:6803/243717916] [v2:172.21.15.53:6804/243717916,v1:172.21.15.53:6805/243717916] exists,up 14c120ad-b975-4c88-b14e-4f72a0d61a3b 2023-07-14T19:24:50.391 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 up in weight 1 up_from 13 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6800/1213220726,v1:172.21.15.70:6801/1213220726] [v2:172.21.15.70:6802/1213220726,v1:172.21.15.70:6803/1213220726] exists,up 301f0944-3cad-4319-9242-4a7e4c4bcc94 2023-07-14T19:24:50.391 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 up in weight 1 up_from 18 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6800/1379504827,v1:172.21.15.104:6801/1379504827] [v2:172.21.15.104:6802/1379504827,v1:172.21.15.104:6803/1379504827] exists,up 7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447 2023-07-14T19:24:50.391 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3 up in weight 1 up_from 24 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6810/1549618174,v1:172.21.15.53:6811/1549618174] [v2:172.21.15.53:6812/1549618174,v1:172.21.15.53:6813/1549618174] exists,up 4b06af11-cdd7-426e-8f3c-68b2b33e750a 2023-07-14T19:24:50.391 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4 up in weight 1 up_from 31 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6808/195242658,v1:172.21.15.70:6809/195242658] [v2:172.21.15.70:6810/195242658,v1:172.21.15.70:6811/195242658] exists,up b7dd3e40-bc97-4894-92ec-21c756a018ed 2023-07-14T19:24:50.391 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5 up in weight 1 up_from 36 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6808/4101602275,v1:172.21.15.104:6809/4101602275] [v2:172.21.15.104:6810/4101602275,v1:172.21.15.104:6811/4101602275] exists,up 530ff111-a77f-4eee-894f-0098aa1febd6 2023-07-14T19:24:50.391 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6 up in weight 1 up_from 43 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6818/4148952318,v1:172.21.15.53:6819/4148952318] [v2:172.21.15.53:6820/4148952318,v1:172.21.15.53:6821/4148952318] exists,up b7ab09b5-1dfc-4d44-a248-7aac58e27749 2023-07-14T19:24:50.391 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7 up in weight 1 up_from 48 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6816/2311839228,v1:172.21.15.70:6817/2311839228] [v2:172.21.15.70:6818/2311839228,v1:172.21.15.70:6819/2311839228] exists,up 3b18c85a-6628-4f12-b07f-d292fcd3a05b 2023-07-14T19:24:50.392 INFO:teuthology.orchestra.run.smithi053.stdout:osd.8 up in weight 1 up_from 56 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6816/48256695,v1:172.21.15.104:6817/48256695] [v2:172.21.15.104:6818/48256695,v1:172.21.15.104:6819/48256695] exists,up 521e5c70-e2f7-45f9-81fc-93bc6401c6b4 2023-07-14T19:24:50.392 INFO:teuthology.orchestra.run.smithi053.stdout:osd.9 up in weight 1 up_from 63 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6826/3378335665,v1:172.21.15.53:6827/3378335665] [v2:172.21.15.53:6828/3378335665,v1:172.21.15.53:6829/3378335665] exists,up c5795443-41f4-4230-9a33-9795ebaffe91 2023-07-14T19:24:50.392 INFO:teuthology.orchestra.run.smithi053.stdout:osd.10 up in weight 1 up_from 68 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6824/1946322503,v1:172.21.15.70:6825/1946322503] [v2:172.21.15.70:6826/1946322503,v1:172.21.15.70:6827/1946322503] exists,up ff27354b-283d-406f-b2ea-fe1c9f769e1a 2023-07-14T19:24:50.392 INFO:teuthology.orchestra.run.smithi053.stdout:osd.11 up in weight 1 up_from 75 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6824/3513976129,v1:172.21.15.104:6825/3513976129] [v2:172.21.15.104:6826/3513976129,v1:172.21.15.104:6827/3513976129] exists,up c2d7cdf0-c8ac-4fc1-a017-e641c666619a 2023-07-14T19:24:50.392 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 2.8 [2,11] 2023-07-14T19:24:50.392 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 2.10 [2,4,8,11] 2023-07-14T19:24:50.392 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 2.1f [8,1] 2023-07-14T19:24:50.393 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 3.1 [4,9] 2023-07-14T19:24:50.393 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 3.17 [4,8] 2023-07-14T19:24:50.393 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 3.1b [5,9] 2023-07-14T19:24:50.393 INFO:teuthology.orchestra.run.smithi053.stdout:pg_upmap_items 3.3c [4,9] 2023-07-14T19:24:50.393 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/1700081881 expires 2023-07-15T19:17:41.240106+0000 2023-07-14T19:24:50.393 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:6801/2149831 expires 2023-07-15T19:17:41.240106+0000 2023-07-14T19:24:50.393 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:6800/2149831 expires 2023-07-15T19:17:41.240106+0000 2023-07-14T19:24:50.393 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/4078421180 expires 2023-07-15T19:17:41.240106+0000 2023-07-14T19:24:50.394 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:6800/3434712469 expires 2023-07-15T19:17:15.286956+0000 2023-07-14T19:24:50.394 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:6801/3434712469 expires 2023-07-15T19:17:15.286956+0000 2023-07-14T19:24:50.394 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/228763847 expires 2023-07-15T19:17:15.286956+0000 2023-07-14T19:24:50.394 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/3718395334 expires 2023-07-15T19:17:15.286956+0000 2023-07-14T19:24:50.394 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/3412815128 expires 2023-07-15T19:17:15.286956+0000 2023-07-14T19:24:50.394 INFO:teuthology.orchestra.run.smithi053.stdout:blocklist 172.21.15.53:0/2206633343 expires 2023-07-15T19:17:41.240106+0000 2023-07-14T19:24:50.395 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f698c0a1260 msgr2=0x7f698c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:50.395 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f698c0a1260 0x7f698c0a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f699400b5c0 tx=0x7f6994005fb0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.395 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00ff8b0 msgr2=0x7f69a019f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:50.396 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00ff8b0 0x7f69a019f330 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6998007ee0 tx=0x7f699800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.396 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 shutdown_connections 2023-07-14T19:24:50.396 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f698c0a1260 0x7f698c0a3710 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.396 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69a00ff8b0 0x7f69a019f330 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.396 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69a00fe5b0 0x7f69a0197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.396 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 --2- 172.21.15.53:0/2393015588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69a00fef00 0x7f69a01982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.396 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.385+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 >> 172.21.15.53:0/2393015588 conn(0x7f69a00fa1e0 msgr2=0x7f69a0106c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:50.396 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.386+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 shutdown_connections 2023-07-14T19:24:50.397 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.386+0000 7f69a8492700 1 -- 172.21.15.53:0/2393015588 wait complete. 2023-07-14T19:24:50.713 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:50 smithi053 ceph-mon[96565]: pgmap v286: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 86 B/s, 1 objects/s recovering 2023-07-14T19:24:50.719 INFO:teuthology.run_tasks:Running task fs.ready... 2023-07-14T19:24:50.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:24:50.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:50 smithi070 ceph-mon[98015]: pgmap v286: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 86 B/s, 1 objects/s recovering 2023-07-14T19:24:50.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:50 smithi104 ceph-mon[98435]: pgmap v286: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 86 B/s, 1 objects/s recovering 2023-07-14T19:24:50.864 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.859+0000 7f962ca78700 1 -- 172.21.15.53:0/4000603391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9628100690 msgr2=0x7f9628106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:50.864 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.859+0000 7f962ca78700 1 --2- 172.21.15.53:0/4000603391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9628100690 0x7f9628106f50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f962000b0a0 tx=0x7f962000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.864 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.859+0000 7f962ca78700 1 -- 172.21.15.53:0/4000603391 shutdown_connections 2023-07-14T19:24:50.865 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.859+0000 7f962ca78700 1 --2- 172.21.15.53:0/4000603391 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9628100690 0x7f9628106f50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.865 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.859+0000 7f962ca78700 1 --2- 172.21.15.53:0/4000603391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96280feb10 0x7f96280fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.865 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.859+0000 7f962ca78700 1 --2- 172.21.15.53:0/4000603391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96280ffd00 0x7f9628100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.865 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.859+0000 7f962ca78700 1 -- 172.21.15.53:0/4000603391 >> 172.21.15.53:0/4000603391 conn(0x7f96280fa130 msgr2=0x7f96280fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:50.866 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.859+0000 7f962ca78700 1 -- 172.21.15.53:0/4000603391 shutdown_connections 2023-07-14T19:24:50.866 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.860+0000 7f962ca78700 1 -- 172.21.15.53:0/4000603391 wait complete. 2023-07-14T19:24:50.866 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.860+0000 7f962ca78700 1 Processor -- start 2023-07-14T19:24:50.866 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.860+0000 7f962ca78700 1 -- start start 2023-07-14T19:24:50.867 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.860+0000 7f962ca78700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96280feb10 0x7f96281959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:50.867 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.860+0000 7f962ca78700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96280ffd00 0x7f9628195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:50.868 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.860+0000 7f962ca78700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9628100690 0x7f962819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:50.868 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.860+0000 7f962ca78700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962819d530 con 0x7f96280ffd00 2023-07-14T19:24:50.868 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.860+0000 7f962ca78700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962819d670 con 0x7f96280feb10 2023-07-14T19:24:50.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.860+0000 7f962ca78700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962819d800 con 0x7f9628100690 2023-07-14T19:24:50.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9625d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96280ffd00 0x7f9628195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:50.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9625d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96280ffd00 0x7f9628195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51230/0 (socket says 172.21.15.53:51230) 2023-07-14T19:24:50.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9625d9b700 1 -- 172.21.15.53:0/303680690 learned_addr learned my addr 172.21.15.53:0/303680690 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:50.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f962659c700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96280feb10 0x7f96281959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:50.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9626d9d700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9628100690 0x7f962819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:50.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9625d9b700 1 -- 172.21.15.53:0/303680690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9628100690 msgr2=0x7f962819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:50.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9625d9b700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9628100690 0x7f962819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9625d9b700 1 -- 172.21.15.53:0/303680690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96280feb10 msgr2=0x7f96281959b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:50.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9625d9b700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96280feb10 0x7f96281959b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:50.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9625d9b700 1 -- 172.21.15.53:0/303680690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620009d00 con 0x7f96280ffd00 2023-07-14T19:24:50.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f9625d9b700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96280ffd00 0x7f9628195ef0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f961800d980 tx=0x7f961800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:50.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f961f7fe700 1 -- 172.21.15.53:0/303680690 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618004220 con 0x7f96280ffd00 2023-07-14T19:24:50.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f961f7fe700 1 -- 172.21.15.53:0/303680690 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9618007640 con 0x7f96280ffd00 2023-07-14T19:24:50.871 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962819dae0 con 0x7f96280ffd00 2023-07-14T19:24:50.871 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f961f7fe700 1 -- 172.21.15.53:0/303680690 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618005690 con 0x7f96280ffd00 2023-07-14T19:24:50.871 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.861+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962819e040 con 0x7f96280ffd00 2023-07-14T19:24:50.871 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.862+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962810a2f0 con 0x7f96280ffd00 2023-07-14T19:24:50.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.863+0000 7f961f7fe700 1 -- 172.21.15.53:0/303680690 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9618004750 con 0x7f96280ffd00 2023-07-14T19:24:50.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.864+0000 7f961f7fe700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96140a12a0 0x7f96140a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:50.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.864+0000 7f962659c700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96140a12a0 0x7f96140a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:50.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.864+0000 7f961f7fe700 1 -- 172.21.15.53:0/303680690 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f96180ccc60 con 0x7f96280ffd00 2023-07-14T19:24:50.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.864+0000 7f962659c700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96140a12a0 0x7f96140a3750 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f96280ffb60 tx=0x7f9610009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:50.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:50.867+0000 7f961f7fe700 1 -- 172.21.15.53:0/303680690 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9618096280 con 0x7f96280ffd00 2023-07-14T19:24:51.070 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.069+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f962810a580 con 0x7f96280ffd00 2023-07-14T19:24:51.072 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.071+0000 7f961f7fe700 1 -- 172.21.15.53:0/303680690 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22459 (secure 0 0 0) 0x7f962810a580 con 0x7f96280ffd00 2023-07-14T19:24:51.073 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:51.073 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:24:42.310475+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:24:51.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96140a12a0 msgr2=0x7f96140a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96140a12a0 0x7f96140a3750 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f96280ffb60 tx=0x7f9610009260 comp rx=0 tx=0).stop 2023-07-14T19:24:51.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96280ffd00 msgr2=0x7f9628195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96280ffd00 0x7f9628195ef0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f961800d980 tx=0x7f961800dc90 comp rx=0 tx=0).stop 2023-07-14T19:24:51.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 shutdown_connections 2023-07-14T19:24:51.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f96140a12a0 0x7f96140a3750 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9628100690 0x7f962819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96280feb10 0x7f96281959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 --2- 172.21.15.53:0/303680690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96280ffd00 0x7f9628195ef0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 >> 172.21.15.53:0/303680690 conn(0x7f96280fa130 msgr2=0x7f9628068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:51.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 shutdown_connections 2023-07-14T19:24:51.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.073+0000 7f962ca78700 1 -- 172.21.15.53:0/303680690 wait complete. 2023-07-14T19:24:51.087 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:19:51.775369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:23:35.935538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:23:37.252372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:24:51.088 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:24:51.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.216+0000 7f6227b47700 1 -- 172.21.15.53:0/4202358059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6220100690 msgr2=0x7f6220106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.225 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.216+0000 7f6227b47700 1 --2- 172.21.15.53:0/4202358059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6220100690 0x7f6220106f50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f621800b0a0 tx=0x7f621800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.216+0000 7f6227b47700 1 -- 172.21.15.53:0/4202358059 shutdown_connections 2023-07-14T19:24:51.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.216+0000 7f6227b47700 1 --2- 172.21.15.53:0/4202358059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6220100690 0x7f6220106f50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.216+0000 7f6227b47700 1 --2- 172.21.15.53:0/4202358059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62200feb10 0x7f62200fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.216+0000 7f6227b47700 1 --2- 172.21.15.53:0/4202358059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62200ffd00 0x7f6220100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.216+0000 7f6227b47700 1 -- 172.21.15.53:0/4202358059 >> 172.21.15.53:0/4202358059 conn(0x7f62200fa130 msgr2=0x7f62200fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:51.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 -- 172.21.15.53:0/4202358059 shutdown_connections 2023-07-14T19:24:51.226 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 -- 172.21.15.53:0/4202358059 wait complete. 2023-07-14T19:24:51.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 Processor -- start 2023-07-14T19:24:51.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 -- start start 2023-07-14T19:24:51.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62200feb10 0x7f6220069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62200ffd00 0x7f622019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6220100690 0x7f622019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622019d560 con 0x7f62200ffd00 2023-07-14T19:24:51.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622019d6d0 con 0x7f62200feb10 2023-07-14T19:24:51.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f6227b47700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622019d890 con 0x7f6220100690 2023-07-14T19:24:51.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f62250e2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62200ffd00 0x7f622019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f62250e2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62200ffd00 0x7f622019a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51244/0 (socket says 172.21.15.53:51244) 2023-07-14T19:24:51.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.217+0000 7f62250e2700 1 -- 172.21.15.53:0/3946926576 learned_addr learned my addr 172.21.15.53:0/3946926576 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:51.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f62258e3700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62200feb10 0x7f6220069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f62260e4700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6220100690 0x7f622019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f62250e2700 1 -- 172.21.15.53:0/3946926576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6220100690 msgr2=0x7f622019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f62250e2700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6220100690 0x7f622019cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f62250e2700 1 -- 172.21.15.53:0/3946926576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62200feb10 msgr2=0x7f6220069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f62250e2700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62200feb10 0x7f6220069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f62250e2700 1 -- 172.21.15.53:0/3946926576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6218009d00 con 0x7f62200ffd00 2023-07-14T19:24:51.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f62250e2700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62200ffd00 0x7f622019a780 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f621000d980 tx=0x7f621000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:51.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f6216ffd700 1 -- 172.21.15.53:0/3946926576 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210004220 con 0x7f62200ffd00 2023-07-14T19:24:51.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f6216ffd700 1 -- 172.21.15.53:0/3946926576 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6210007640 con 0x7f62200ffd00 2023-07-14T19:24:51.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622019db70 con 0x7f62200ffd00 2023-07-14T19:24:51.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f6216ffd700 1 -- 172.21.15.53:0/3946926576 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210005610 con 0x7f62200ffd00 2023-07-14T19:24:51.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.218+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622019e0d0 con 0x7f62200ffd00 2023-07-14T19:24:51.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.219+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62201059a0 con 0x7f62200ffd00 2023-07-14T19:24:51.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.221+0000 7f6216ffd700 1 -- 172.21.15.53:0/3946926576 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6210004750 con 0x7f62200ffd00 2023-07-14T19:24:51.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.221+0000 7f6216ffd700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f620c0a1260 0x7f620c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.221+0000 7f6216ffd700 1 -- 172.21.15.53:0/3946926576 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f62100ccd20 con 0x7f62200ffd00 2023-07-14T19:24:51.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.221+0000 7f62258e3700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f620c0a1260 0x7f620c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.222+0000 7f62258e3700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f620c0a1260 0x7f620c0a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f62200ffb60 tx=0x7f621c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:51.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.225+0000 7f6216ffd700 1 -- 172.21.15.53:0/3946926576 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6210096340 con 0x7f62200ffd00 2023-07-14T19:24:51.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.424+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f622002cb10 con 0x7f62200ffd00 2023-07-14T19:24:51.425 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.424+0000 7f6216ffd700 1 -- 172.21.15.53:0/3946926576 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6210016020 con 0x7f62200ffd00 2023-07-14T19:24:51.426 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2023-07-14T19:24:51.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f620c0a1260 msgr2=0x7f620c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f620c0a1260 0x7f620c0a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f62200ffb60 tx=0x7f621c009260 comp rx=0 tx=0).stop 2023-07-14T19:24:51.427 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62200ffd00 msgr2=0x7f622019a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62200ffd00 0x7f622019a780 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f621000d980 tx=0x7f621000dc90 comp rx=0 tx=0).stop 2023-07-14T19:24:51.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 shutdown_connections 2023-07-14T19:24:51.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f620c0a1260 0x7f620c0a3710 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6220100690 0x7f622019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62200feb10 0x7f6220069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 --2- 172.21.15.53:0/3946926576 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62200ffd00 0x7f622019a780 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.428 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 >> 172.21.15.53:0/3946926576 conn(0x7f62200fa130 msgr2=0x7f62200fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:51.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 shutdown_connections 2023-07-14T19:24:51.429 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.426+0000 7f6227b47700 1 -- 172.21.15.53:0/3946926576 wait complete. 2023-07-14T19:24:51.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:24:51.568 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.567+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/2492724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d84103010 msgr2=0x7f5d841053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.567+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/2492724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d84103010 0x7f5d841053f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009990 tx=0x7f5d74009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.567+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/2492724 shutdown_connections 2023-07-14T19:24:51.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.567+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/2492724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d84105930 0x7f5d84107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.567+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/2492724 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d841006f0 0x7f5d84102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.567+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/2492724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d84103010 0x7f5d841053f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.569 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.567+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/2492724 >> 172.21.15.53:0/2492724 conn(0x7f5d840fa130 msgr2=0x7f5d840fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:51.570 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.567+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/2492724 shutdown_connections 2023-07-14T19:24:51.570 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.567+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/2492724 wait complete. 2023-07-14T19:24:51.571 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8c0fd700 1 Processor -- start 2023-07-14T19:24:51.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8c0fd700 1 -- start start 2023-07-14T19:24:51.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8c0fd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d841006f0 0x7f5d84100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8c0fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d84103010 0x7f5d840fe6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8c0fd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d84105930 0x7f5d840febe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8c0fd700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840ff380 con 0x7f5d841006f0 2023-07-14T19:24:51.572 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8c0fd700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d840ff4f0 con 0x7f5d84105930 2023-07-14T19:24:51.573 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8c0fd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8419f9b0 con 0x7f5d84103010 2023-07-14T19:24:51.573 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89e99700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d841006f0 0x7f5d84100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.573 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89e99700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d841006f0 0x7f5d84100060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51256/0 (socket says 172.21.15.53:51256) 2023-07-14T19:24:51.573 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89e99700 1 -- 172.21.15.53:0/3647058819 learned_addr learned my addr 172.21.15.53:0/3647058819 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:51.573 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89698700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d84103010 0x7f5d840fe6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.573 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8a69a700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d84105930 0x7f5d840febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.573 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89e99700 1 -- 172.21.15.53:0/3647058819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d84103010 msgr2=0x7f5d840fe6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.574 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89e99700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d84103010 0x7f5d840fe6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.574 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89e99700 1 -- 172.21.15.53:0/3647058819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d84105930 msgr2=0x7f5d840febe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.574 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89e99700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d84105930 0x7f5d840febe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.574 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89e99700 1 -- 172.21.15.53:0/3647058819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d740095f0 con 0x7f5d841006f0 2023-07-14T19:24:51.574 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d8a69a700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d84105930 0x7f5d840febe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:51.574 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.568+0000 7f5d89e99700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d841006f0 0x7f5d84100060 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000e9a0 tx=0x7f5d8000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:51.574 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.569+0000 7f5d7affd700 1 -- 172.21.15.53:0/3647058819 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80009790 con 0x7f5d841006f0 2023-07-14T19:24:51.575 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.569+0000 7f5d7affd700 1 -- 172.21.15.53:0/3647058819 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d8000cbd0 con 0x7f5d841006f0 2023-07-14T19:24:51.575 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.569+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8419fc90 con 0x7f5d841006f0 2023-07-14T19:24:51.575 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.569+0000 7f5d7affd700 1 -- 172.21.15.53:0/3647058819 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80018680 con 0x7f5d841006f0 2023-07-14T19:24:51.575 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.569+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d841a01a0 con 0x7f5d841006f0 2023-07-14T19:24:51.575 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.570+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d840fbca0 con 0x7f5d841006f0 2023-07-14T19:24:51.575 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.571+0000 7f5d7affd700 1 -- 172.21.15.53:0/3647058819 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5d8000cd40 con 0x7f5d841006f0 2023-07-14T19:24:51.576 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.571+0000 7f5d7affd700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d700a11e0 0x7f5d700a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.576 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.571+0000 7f5d89698700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d700a11e0 0x7f5d700a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.576 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.571+0000 7f5d7affd700 1 -- 172.21.15.53:0/3647058819 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f5d80014070 con 0x7f5d841006f0 2023-07-14T19:24:51.577 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.572+0000 7f5d89698700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d700a11e0 0x7f5d700a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400b2e0 tx=0x7f5d74005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:51.577 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.575+0000 7f5d7affd700 1 -- 172.21.15.53:0/3647058819 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5d80051780 con 0x7f5d841006f0 2023-07-14T19:24:51.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:51 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2393015588' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:24:51.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:51 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/303680690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:51.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:51 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2393015588' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:24:51.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:51 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/303680690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:51.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.774+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5d8402cb60 con 0x7f5d841006f0 2023-07-14T19:24:51.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.775+0000 7f5d7affd700 1 -- 172.21.15.53:0/3647058819 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5d80096330 con 0x7f5d841006f0 2023-07-14T19:24:51.777 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2023-07-14T19:24:51.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d700a11e0 msgr2=0x7f5d700a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d700a11e0 0x7f5d700a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400b2e0 tx=0x7f5d74005fb0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d841006f0 msgr2=0x7f5d84100060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d841006f0 0x7f5d84100060 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000e9a0 tx=0x7f5d8000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 shutdown_connections 2023-07-14T19:24:51.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f5d700a11e0 0x7f5d700a3690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d84105930 0x7f5d840febe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d841006f0 0x7f5d84100060 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 --2- 172.21.15.53:0/3647058819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d84103010 0x7f5d840fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 >> 172.21.15.53:0/3647058819 conn(0x7f5d840fa130 msgr2=0x7f5d84101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:51.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 shutdown_connections 2023-07-14T19:24:51.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.777+0000 7f5d8c0fd700 1 -- 172.21.15.53:0/3647058819 wait complete. 2023-07-14T19:24:51.789 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:51 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2393015588' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:24:51.790 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:51 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/303680690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:51.791 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:24:51.919 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.918+0000 7fe1a8269700 1 -- 172.21.15.53:0/3327803410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 msgr2=0x7fe1a00fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.918+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3327803410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 0x7fe1a00fef20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe190009990 tx=0x7fe190009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.918+0000 7fe1a8269700 1 -- 172.21.15.53:0/3327803410 shutdown_connections 2023-07-14T19:24:51.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.918+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3327803410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a0100690 0x7fe1a0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.918+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3327803410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 0x7fe1a00fef20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.920 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.918+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3327803410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1a00ffd00 0x7fe1a0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.921 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.918+0000 7fe1a8269700 1 -- 172.21.15.53:0/3327803410 >> 172.21.15.53:0/3327803410 conn(0x7fe1a00fa130 msgr2=0x7fe1a00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:51.921 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.918+0000 7fe1a8269700 1 -- 172.21.15.53:0/3327803410 shutdown_connections 2023-07-14T19:24:51.921 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.918+0000 7fe1a8269700 1 -- 172.21.15.53:0/3327803410 wait complete. 2023-07-14T19:24:51.922 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a8269700 1 Processor -- start 2023-07-14T19:24:51.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a8269700 1 -- start start 2023-07-14T19:24:51.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a8269700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 0x7fe1a0102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a8269700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1a00ffd00 0x7fe1a01010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a8269700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a0100690 0x7fe1a0101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.923 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a8269700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a0101dc0 con 0x7fe1a00feb10 2023-07-14T19:24:51.924 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a8269700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a0101f30 con 0x7fe1a0100690 2023-07-14T19:24:51.924 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a8269700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a019f960 con 0x7fe1a00ffd00 2023-07-14T19:24:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6005700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 0x7fe1a0102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6005700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 0x7fe1a0102aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51270/0 (socket says 172.21.15.53:51270) 2023-07-14T19:24:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6005700 1 -- 172.21.15.53:0/3179779960 learned_addr learned my addr 172.21.15.53:0/3179779960 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6806700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a0100690 0x7fe1a0101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a5804700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1a00ffd00 0x7fe1a01010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6005700 1 -- 172.21.15.53:0/3179779960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1a00ffd00 msgr2=0x7fe1a01010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6005700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1a00ffd00 0x7fe1a01010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6005700 1 -- 172.21.15.53:0/3179779960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a0100690 msgr2=0x7fe1a0101620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6005700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a0100690 0x7fe1a0101620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6005700 1 -- 172.21.15.53:0/3179779960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1900095f0 con 0x7fe1a00feb10 2023-07-14T19:24:51.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a5804700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1a00ffd00 0x7fe1a01010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:51.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.919+0000 7fe1a6005700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 0x7fe1a0102aa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe190000c00 tx=0x7fe1900057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:51.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.920+0000 7fe1977fe700 1 -- 172.21.15.53:0/3179779960 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19001d070 con 0x7fe1a00feb10 2023-07-14T19:24:51.928 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.920+0000 7fe1977fe700 1 -- 172.21.15.53:0/3179779960 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe19000fc40 con 0x7fe1a00feb10 2023-07-14T19:24:51.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.920+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a019fbe0 con 0x7fe1a00feb10 2023-07-14T19:24:51.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.920+0000 7fe1977fe700 1 -- 172.21.15.53:0/3179779960 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190017650 con 0x7fe1a00feb10 2023-07-14T19:24:51.929 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.920+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a01a0090 con 0x7fe1a00feb10 2023-07-14T19:24:51.930 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.921+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a010a2e0 con 0x7fe1a00feb10 2023-07-14T19:24:51.930 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.922+0000 7fe1977fe700 1 -- 172.21.15.53:0/3179779960 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe190021410 con 0x7fe1a00feb10 2023-07-14T19:24:51.931 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.922+0000 7fe1977fe700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe18c0a11e0 0x7fe18c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:51.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.922+0000 7fe1a5804700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe18c0a11e0 0x7fe18c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:51.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.922+0000 7fe1977fe700 1 -- 172.21.15.53:0/3179779960 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fe1900ce6c0 con 0x7fe1a00feb10 2023-07-14T19:24:51.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.923+0000 7fe1a5804700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe18c0a11e0 0x7fe18c0a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe19c005fd0 tx=0x7fe19c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:51.932 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:51.926+0000 7fe1977fe700 1 -- 172.21.15.53:0/3179779960 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe190097d70 con 0x7fe1a00feb10 2023-07-14T19:24:52.125 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.124+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe1a01a0640 con 0x7fe1a00feb10 2023-07-14T19:24:52.126 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.125+0000 7fe1977fe700 1 -- 172.21.15.53:0/3179779960 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe19009b390 con 0x7fe1a00feb10 2023-07-14T19:24:52.126 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2023-07-14T19:24:52.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe18c0a11e0 msgr2=0x7fe18c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe18c0a11e0 0x7fe18c0a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe19c005fd0 tx=0x7fe19c005e20 comp rx=0 tx=0).stop 2023-07-14T19:24:52.128 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 msgr2=0x7fe1a0102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 0x7fe1a0102aa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe190000c00 tx=0x7fe1900057c0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 shutdown_connections 2023-07-14T19:24:52.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fe18c0a11e0 0x7fe18c0a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a0100690 0x7fe1a0101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a00feb10 0x7fe1a0102aa0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.129 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 --2- 172.21.15.53:0/3179779960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1a00ffd00 0x7fe1a01010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 >> 172.21.15.53:0/3179779960 conn(0x7fe1a00fa130 msgr2=0x7fe1a0067780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:52.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 shutdown_connections 2023-07-14T19:24:52.130 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.127+0000 7fe1a8269700 1 -- 172.21.15.53:0/3179779960 wait complete. 2023-07-14T19:24:52.141 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:24:52.274 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 -- 172.21.15.53:0/2936664732 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c103040 msgr2=0x7f170c105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 --2- 172.21.15.53:0/2936664732 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c103040 0x7f170c105420 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009990 tx=0x7f16f4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 -- 172.21.15.53:0/2936664732 shutdown_connections 2023-07-14T19:24:52.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 --2- 172.21.15.53:0/2936664732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f170c105960 0x7f170c107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 --2- 172.21.15.53:0/2936664732 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f170c100720 0x7f170c102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.275 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 --2- 172.21.15.53:0/2936664732 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c103040 0x7f170c105420 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 -- 172.21.15.53:0/2936664732 >> 172.21.15.53:0/2936664732 conn(0x7f170c0fa100 msgr2=0x7f170c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:52.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 -- 172.21.15.53:0/2936664732 shutdown_connections 2023-07-14T19:24:52.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 -- 172.21.15.53:0/2936664732 wait complete. 2023-07-14T19:24:52.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 Processor -- start 2023-07-14T19:24:52.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.271+0000 7f1712859700 1 -- start start 2023-07-14T19:24:52.276 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1712859700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f170c100720 0x7f170c100090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:52.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1712859700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f170c103040 0x7f170c0fe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:52.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1712859700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c105960 0x7f170c0fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:52.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1712859700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0ff3b0 con 0x7f170c105960 2023-07-14T19:24:52.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1712859700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0ff520 con 0x7f170c103040 2023-07-14T19:24:52.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1712859700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c19f8d0 con 0x7f170c100720 2023-07-14T19:24:52.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1710df6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c105960 0x7f170c0fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:52.277 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1710df6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c105960 0x7f170c0fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51288/0 (socket says 172.21.15.53:51288) 2023-07-14T19:24:52.278 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1710df6700 1 -- 172.21.15.53:0/3725529666 learned_addr learned my addr 172.21.15.53:0/3725529666 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:52.278 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f170bfff700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f170c100720 0x7f170c100090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:52.278 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f170b7fe700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f170c103040 0x7f170c0fe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:52.278 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1710df6700 1 -- 172.21.15.53:0/3725529666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f170c100720 msgr2=0x7f170c100090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.278 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1710df6700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f170c100720 0x7f170c100090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1710df6700 1 -- 172.21.15.53:0/3725529666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f170c103040 msgr2=0x7f170c0fe6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1710df6700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f170c103040 0x7f170c0fe6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1710df6700 1 -- 172.21.15.53:0/3725529666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f40095f0 con 0x7f170c105960 2023-07-14T19:24:52.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f170b7fe700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f170c103040 0x7f170c0fe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:52.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f1710df6700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c105960 0x7f170c0fec10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1704007ee0 tx=0x7f170400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:52.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.272+0000 7f17097fa700 1 -- 172.21.15.53:0/3725529666 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704017070 con 0x7f170c105960 2023-07-14T19:24:52.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.273+0000 7f17097fa700 1 -- 172.21.15.53:0/3725529666 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f170400f040 con 0x7f170c105960 2023-07-14T19:24:52.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.273+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c19fbb0 con 0x7f170c105960 2023-07-14T19:24:52.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.273+0000 7f17097fa700 1 -- 172.21.15.53:0/3725529666 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704005740 con 0x7f170c105960 2023-07-14T19:24:52.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.273+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c1a0070 con 0x7f170c105960 2023-07-14T19:24:52.280 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.274+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c0fbc70 con 0x7f170c105960 2023-07-14T19:24:52.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.275+0000 7f17097fa700 1 -- 172.21.15.53:0/3725529666 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f170400dc60 con 0x7f170c105960 2023-07-14T19:24:52.281 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.276+0000 7f17097fa700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f16f80a1220 0x7f16f80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:52.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.276+0000 7f17097fa700 1 -- 172.21.15.53:0/3725529666 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f17040cc8c0 con 0x7f170c105960 2023-07-14T19:24:52.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.276+0000 7f170bfff700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f16f80a1220 0x7f16f80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:52.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.276+0000 7f170bfff700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f16f80a1220 0x7f16f80a36d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00ac50 tx=0x7f16fc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:52.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.279+0000 7f17097fa700 1 -- 172.21.15.53:0/3725529666 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1704095ee0 con 0x7f170c105960 2023-07-14T19:24:52.528 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.527+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f170c02caf0 con 0x7f170c105960 2023-07-14T19:24:52.530 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.528+0000 7f17097fa700 1 -- 172.21.15.53:0/3725529666 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f1704099500 con 0x7f170c105960 2023-07-14T19:24:52.530 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:52.530 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:23:37.894565+0000","modified":"2023-07-14T19:23:43.289341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24491,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24484":{"gid":24484,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24491":{"gid":24491,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6839/2496341573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":2496341573},{"type":"v1","addr":"172.21.15.70:6839","nonce":2496341573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:24:52.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f16f80a1220 msgr2=0x7f16f80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.532 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f16f80a1220 0x7f16f80a36d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00ac50 tx=0x7f16fc00a380 comp rx=0 tx=0).stop 2023-07-14T19:24:52.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c105960 msgr2=0x7f170c0fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c105960 0x7f170c0fec10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1704007ee0 tx=0x7f170400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.533 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 shutdown_connections 2023-07-14T19:24:52.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f16f80a1220 0x7f16f80a36d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f170c105960 0x7f170c0fec10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f170c100720 0x7f170c100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.534 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 --2- 172.21.15.53:0/3725529666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f170c103040 0x7f170c0fe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 >> 172.21.15.53:0/3725529666 conn(0x7f170c0fa100 msgr2=0x7f170c101380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:52.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 shutdown_connections 2023-07-14T19:24:52.535 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.530+0000 7f1712859700 1 -- 172.21.15.53:0/3725529666 wait complete. 2023-07-14T19:24:52.536 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 23 2023-07-14T19:24:52.543 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:52 smithi053 ceph-mon[96565]: pgmap v287: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 69 B/s, 1 objects/s recovering 2023-07-14T19:24:52.543 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:52 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3946926576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:52.543 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:52 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3647058819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:52.543 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:52 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3179779960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:52.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:24:52.671 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.668+0000 7fd4ba281700 1 -- 172.21.15.53:0/572984824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b41006b0 msgr2=0x7fd4b4106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.671 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.668+0000 7fd4ba281700 1 --2- 172.21.15.53:0/572984824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b41006b0 0x7fd4b4106f70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00b0a0 tx=0x7fd4ac00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.672 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.668+0000 7fd4ba281700 1 -- 172.21.15.53:0/572984824 shutdown_connections 2023-07-14T19:24:52.672 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.668+0000 7fd4ba281700 1 --2- 172.21.15.53:0/572984824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b41006b0 0x7fd4b4106f70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.672 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.668+0000 7fd4ba281700 1 --2- 172.21.15.53:0/572984824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4b40feb30 0x7fd4b40fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.672 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.668+0000 7fd4ba281700 1 --2- 172.21.15.53:0/572984824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4b40ffd20 0x7fd4b4100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.672 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.668+0000 7fd4ba281700 1 -- 172.21.15.53:0/572984824 >> 172.21.15.53:0/572984824 conn(0x7fd4b40fa110 msgr2=0x7fd4b40fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:52.672 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.668+0000 7fd4ba281700 1 -- 172.21.15.53:0/572984824 shutdown_connections 2023-07-14T19:24:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.668+0000 7fd4ba281700 1 -- 172.21.15.53:0/572984824 wait complete. 2023-07-14T19:24:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4ba281700 1 Processor -- start 2023-07-14T19:24:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4ba281700 1 -- start start 2023-07-14T19:24:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4ba281700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b40feb30 0x7fd4b4102ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4ba281700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4b40ffd20 0x7fd4b4101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4ba281700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4b41006b0 0x7fd4b4101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4ba281700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b4101de0 con 0x7fd4b40feb30 2023-07-14T19:24:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4ba281700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b4101f50 con 0x7fd4b40ffd20 2023-07-14T19:24:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4ba281700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b419fa40 con 0x7fd4b41006b0 2023-07-14T19:24:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b40feb30 0x7fd4b4102ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b3fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b40feb30 0x7fd4b4102ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51298/0 (socket says 172.21.15.53:51298) 2023-07-14T19:24:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b3fff700 1 -- 172.21.15.53:0/3079288879 learned_addr learned my addr 172.21.15.53:0/3079288879 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b881e700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4b41006b0 0x7fd4b4101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b37fe700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4b40ffd20 0x7fd4b4101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b3fff700 1 -- 172.21.15.53:0/3079288879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4b41006b0 msgr2=0x7fd4b4101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.675 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b3fff700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4b41006b0 0x7fd4b4101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.675 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b3fff700 1 -- 172.21.15.53:0/3079288879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4b40ffd20 msgr2=0x7fd4b4101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.675 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b3fff700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4b40ffd20 0x7fd4b4101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.675 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.669+0000 7fd4b3fff700 1 -- 172.21.15.53:0/3079288879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac009d00 con 0x7fd4b40feb30 2023-07-14T19:24:52.675 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.670+0000 7fd4b3fff700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b40feb30 0x7fd4b4102ac0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400d980 tx=0x7fd4a400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:52.675 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.670+0000 7fd4b17fa700 1 -- 172.21.15.53:0/3079288879 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a400de60 con 0x7fd4b40feb30 2023-07-14T19:24:52.675 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.670+0000 7fd4b17fa700 1 -- 172.21.15.53:0/3079288879 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4a40044e0 con 0x7fd4b40feb30 2023-07-14T19:24:52.676 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.670+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b419fd20 con 0x7fd4b40feb30 2023-07-14T19:24:52.676 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.670+0000 7fd4b17fa700 1 -- 172.21.15.53:0/3079288879 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a4003be0 con 0x7fd4b40feb30 2023-07-14T19:24:52.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.670+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b41a0180 con 0x7fd4b40feb30 2023-07-14T19:24:52.677 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.671+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b410a310 con 0x7fd4b40feb30 2023-07-14T19:24:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.672+0000 7fd4b17fa700 1 -- 172.21.15.53:0/3079288879 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd4a4009bb0 con 0x7fd4b40feb30 2023-07-14T19:24:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.673+0000 7fd4b17fa700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd49c0a1220 0x7fd49c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.673+0000 7fd4b37fe700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd49c0a1220 0x7fd49c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.673+0000 7fd4b17fa700 1 -- 172.21.15.53:0/3079288879 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fd4a40ccc10 con 0x7fd4b40feb30 2023-07-14T19:24:52.679 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.673+0000 7fd4b37fe700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd49c0a1220 0x7fd49c0a36d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8005fd0 tx=0x7fd4a800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.676+0000 7fd4b17fa700 1 -- 172.21.15.53:0/3079288879 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd4a40961d0 con 0x7fd4b40feb30 2023-07-14T19:24:52.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:52 smithi070 ceph-mon[98015]: pgmap v287: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 69 B/s, 1 objects/s recovering 2023-07-14T19:24:52.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:52 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3946926576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:52.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:52 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3647058819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:52.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:52 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3179779960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:52.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:52 smithi104 ceph-mon[98435]: pgmap v287: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 69 B/s, 1 objects/s recovering 2023-07-14T19:24:52.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:52 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3946926576' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:52.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:52 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3647058819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:52.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:52 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3179779960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:52.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.876+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4b41a0820 con 0x7fd4b40feb30 2023-07-14T19:24:52.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.878+0000 7fd4b17fa700 1 -- 172.21.15.53:0/3079288879 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22459 (secure 0 0 0) 0x7fd4a4009940 con 0x7fd4b40feb30 2023-07-14T19:24:52.879 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:52.880 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:24:42.310475+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:24:52.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd49c0a1220 msgr2=0x7fd49c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd49c0a1220 0x7fd49c0a36d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8005fd0 tx=0x7fd4a800a560 comp rx=0 tx=0).stop 2023-07-14T19:24:52.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b40feb30 msgr2=0x7fd4b4102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:52.882 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b40feb30 0x7fd4b4102ac0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400d980 tx=0x7fd4a400dd40 comp rx=0 tx=0).stop 2023-07-14T19:24:52.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 shutdown_connections 2023-07-14T19:24:52.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fd49c0a1220 0x7fd49c0a36d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd4b41006b0 0x7fd4b4101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.883 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4b40feb30 0x7fd4b4102ac0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.884 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 --2- 172.21.15.53:0/3079288879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4b40ffd20 0x7fd4b4101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:52.884 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 >> 172.21.15.53:0/3079288879 conn(0x7fd4b40fa110 msgr2=0x7fd4b4068830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:52.884 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 shutdown_connections 2023-07-14T19:24:52.885 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:52.880+0000 7fd4ba281700 1 -- 172.21.15.53:0/3079288879 wait complete. 2023-07-14T19:24:52.894 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:19:51.775369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:23:35.935538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:23:37.252372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:24:52.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:24:53.024 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.023+0000 7f8c2fe15700 1 -- 172.21.15.53:0/3906398582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280ffd00 msgr2=0x7f8c28100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.023+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/3906398582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280ffd00 0x7f8c28100150 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009990 tx=0x7f8c18009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.023+0000 7f8c2fe15700 1 -- 172.21.15.53:0/3906398582 shutdown_connections 2023-07-14T19:24:53.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.023+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/3906398582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c28100690 0x7f8c28106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.023+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/3906398582 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c280feb10 0x7f8c280fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.023+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/3906398582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280ffd00 0x7f8c28100150 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.023+0000 7f8c2fe15700 1 -- 172.21.15.53:0/3906398582 >> 172.21.15.53:0/3906398582 conn(0x7f8c280fa130 msgr2=0x7f8c280fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:53.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.023+0000 7f8c2fe15700 1 -- 172.21.15.53:0/3906398582 shutdown_connections 2023-07-14T19:24:53.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.023+0000 7f8c2fe15700 1 -- 172.21.15.53:0/3906398582 wait complete. 2023-07-14T19:24:53.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2fe15700 1 Processor -- start 2023-07-14T19:24:53.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2fe15700 1 -- start start 2023-07-14T19:24:53.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2fe15700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280feb10 0x7f8c28068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2fe15700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c280ffd00 0x7f8c2819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2fe15700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c28100690 0x7f8c2819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2fe15700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2819d4f0 con 0x7f8c280feb10 2023-07-14T19:24:53.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2fe15700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2819d660 con 0x7f8c280ffd00 2023-07-14T19:24:53.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2fe15700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c2819d820 con 0x7f8c28100690 2023-07-14T19:24:53.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2dbb1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280feb10 0x7f8c28068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2dbb1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280feb10 0x7f8c28068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51312/0 (socket says 172.21.15.53:51312) 2023-07-14T19:24:53.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2dbb1700 1 -- 172.21.15.53:0/4085621114 learned_addr learned my addr 172.21.15.53:0/4085621114 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:53.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2d3b0700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c280ffd00 0x7f8c2819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2e3b2700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c28100690 0x7f8c2819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2dbb1700 1 -- 172.21.15.53:0/4085621114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c28100690 msgr2=0x7f8c2819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2dbb1700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c28100690 0x7f8c2819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2dbb1700 1 -- 172.21.15.53:0/4085621114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c280ffd00 msgr2=0x7f8c2819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2dbb1700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c280ffd00 0x7f8c2819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2dbb1700 1 -- 172.21.15.53:0/4085621114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c20009c40 con 0x7f8c280feb10 2023-07-14T19:24:53.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c2dbb1700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280feb10 0x7f8c28068310 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400e970 tx=0x7f8c2400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:53.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.024+0000 7f8c1effd700 1 -- 172.21.15.53:0/4085621114 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c24009790 con 0x7f8c280feb10 2023-07-14T19:24:53.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.025+0000 7f8c1effd700 1 -- 172.21.15.53:0/4085621114 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c2400eec0 con 0x7f8c280feb10 2023-07-14T19:24:53.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.025+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c180095f0 con 0x7f8c280feb10 2023-07-14T19:24:53.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.025+0000 7f8c1effd700 1 -- 172.21.15.53:0/4085621114 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c24003cf0 con 0x7f8c280feb10 2023-07-14T19:24:53.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.025+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c2819de80 con 0x7f8c280feb10 2023-07-14T19:24:53.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.025+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c28105a90 con 0x7f8c280feb10 2023-07-14T19:24:53.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.027+0000 7f8c1effd700 1 -- 172.21.15.53:0/4085621114 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8c2400ccd0 con 0x7f8c280feb10 2023-07-14T19:24:53.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.028+0000 7f8c1effd700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8c140a11e0 0x7f8c140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.028+0000 7f8c2d3b0700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8c140a11e0 0x7f8c140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.028+0000 7f8c1effd700 1 -- 172.21.15.53:0/4085621114 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f8c24014070 con 0x7f8c280feb10 2023-07-14T19:24:53.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.028+0000 7f8c2d3b0700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8c140a11e0 0x7f8c140a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8c18005b00 tx=0x7f8c18005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:53.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.031+0000 7f8c1effd700 1 -- 172.21.15.53:0/4085621114 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8c24096420 con 0x7f8c280feb10 2023-07-14T19:24:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.234+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c2819d9e0 con 0x7f8c280feb10 2023-07-14T19:24:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.234+0000 7f8c1effd700 1 -- 172.21.15.53:0/4085621114 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c24099a40 con 0x7f8c280feb10 2023-07-14T19:24:53.241 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2023-07-14T19:24:53.242 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8c140a11e0 msgr2=0x7f8c140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.242 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8c140a11e0 0x7f8c140a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8c18005b00 tx=0x7f8c18005a90 comp rx=0 tx=0).stop 2023-07-14T19:24:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280feb10 msgr2=0x7f8c28068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280feb10 0x7f8c28068310 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400e970 tx=0x7f8c2400ec80 comp rx=0 tx=0).stop 2023-07-14T19:24:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 shutdown_connections 2023-07-14T19:24:53.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8c140a11e0 0x7f8c140a3690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8c28100690 0x7f8c2819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c280feb10 0x7f8c28068310 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.244 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 --2- 172.21.15.53:0/4085621114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c280ffd00 0x7f8c2819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.245 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 >> 172.21.15.53:0/4085621114 conn(0x7f8c280fa130 msgr2=0x7f8c280fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:53.245 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 shutdown_connections 2023-07-14T19:24:53.245 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.236+0000 7f8c2fe15700 1 -- 172.21.15.53:0/4085621114 wait complete. 2023-07-14T19:24:53.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:24:53.382 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.379+0000 7f3576fbc700 1 -- 172.21.15.53:0/2093084235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3570100690 msgr2=0x7f3570106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.383 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.379+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2093084235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3570100690 0x7f3570106f50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f356800b0a0 tx=0x7f356800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.383 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.379+0000 7f3576fbc700 1 -- 172.21.15.53:0/2093084235 shutdown_connections 2023-07-14T19:24:53.383 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.379+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2093084235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3570100690 0x7f3570106f50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.383 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.379+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2093084235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35700feb10 0x7f35700fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.383 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.379+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2093084235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700ffd00 0x7f3570100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.383 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.379+0000 7f3576fbc700 1 -- 172.21.15.53:0/2093084235 >> 172.21.15.53:0/2093084235 conn(0x7f35700fa130 msgr2=0x7f35700fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:53.384 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.379+0000 7f3576fbc700 1 -- 172.21.15.53:0/2093084235 shutdown_connections 2023-07-14T19:24:53.384 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.379+0000 7f3576fbc700 1 -- 172.21.15.53:0/2093084235 wait complete. 2023-07-14T19:24:53.384 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3576fbc700 1 Processor -- start 2023-07-14T19:24:53.385 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3576fbc700 1 -- start start 2023-07-14T19:24:53.385 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3576fbc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700feb10 0x7f3570197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.385 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3576fbc700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700ffd00 0x7f35701981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.385 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3576fbc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570100690 0x7f357019f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.386 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3576fbc700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357019f7f0 con 0x7f35700ffd00 2023-07-14T19:24:53.386 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3576fbc700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357019f960 con 0x7f3570100690 2023-07-14T19:24:53.386 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3576fbc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357019fb20 con 0x7f35700feb10 2023-07-14T19:24:53.386 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f356ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700ffd00 0x7f35701981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.387 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f356ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700ffd00 0x7f35701981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51326/0 (socket says 172.21.15.53:51326) 2023-07-14T19:24:53.387 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f356ffff700 1 -- 172.21.15.53:0/2555602464 learned_addr learned my addr 172.21.15.53:0/2555602464 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:53.387 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3574d58700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700feb10 0x7f3570197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.387 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.380+0000 7f3575559700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570100690 0x7f357019f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.387 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f356ffff700 1 -- 172.21.15.53:0/2555602464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700feb10 msgr2=0x7f3570197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.388 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f356ffff700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700feb10 0x7f3570197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.388 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f356ffff700 1 -- 172.21.15.53:0/2555602464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570100690 msgr2=0x7f357019f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.388 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f356ffff700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570100690 0x7f357019f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.388 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f356ffff700 1 -- 172.21.15.53:0/2555602464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3568009d00 con 0x7f35700ffd00 2023-07-14T19:24:53.388 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f3574d58700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700feb10 0x7f3570197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:53.388 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f3575559700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570100690 0x7f357019f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f356ffff700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700ffd00 0x7f35701981c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f355c00e9a0 tx=0x7f355c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f356dffb700 1 -- 172.21.15.53:0/2555602464 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c009790 con 0x7f35700ffd00 2023-07-14T19:24:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f356dffb700 1 -- 172.21.15.53:0/2555602464 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f355c00cbd0 con 0x7f35700ffd00 2023-07-14T19:24:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357019fe00 con 0x7f35700ffd00 2023-07-14T19:24:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f356dffb700 1 -- 172.21.15.53:0/2555602464 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c018680 con 0x7f35700ffd00 2023-07-14T19:24:53.389 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.381+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35701a0290 con 0x7f35700ffd00 2023-07-14T19:24:53.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.382+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570106080 con 0x7f35700ffd00 2023-07-14T19:24:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.383+0000 7f356dffb700 1 -- 172.21.15.53:0/2555602464 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f355c00cd40 con 0x7f35700ffd00 2023-07-14T19:24:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.384+0000 7f356dffb700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35600a1240 0x7f35600a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.384+0000 7f356dffb700 1 -- 172.21.15.53:0/2555602464 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f355c014070 con 0x7f35700ffd00 2023-07-14T19:24:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.384+0000 7f3574d58700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35600a1240 0x7f35600a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.392 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.384+0000 7f3574d58700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35600a1240 0x7f35600a36f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3564005950 tx=0x7f35640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:53.392 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.387+0000 7f356dffb700 1 -- 172.21.15.53:0/2555602464 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f355c0967d0 con 0x7f35700ffd00 2023-07-14T19:24:53.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:53 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3725529666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:53.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:53 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3079288879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:53.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:53 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/4085621114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:53.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:53 smithi053 ceph-mon[96565]: pgmap v288: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 63 B/s, 1 objects/s recovering 2023-07-14T19:24:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.589+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f357002cb30 con 0x7f35700ffd00 2023-07-14T19:24:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.590+0000 7f356dffb700 1 -- 172.21.15.53:0/2555602464 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f355c099df0 con 0x7f35700ffd00 2023-07-14T19:24:53.592 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2023-07-14T19:24:53.594 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35600a1240 msgr2=0x7f35600a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.594 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35600a1240 0x7f35600a36f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3564005950 tx=0x7f35640058e0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.594 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700ffd00 msgr2=0x7f35701981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.594 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700ffd00 0x7f35701981c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f355c00e9a0 tx=0x7f355c00ed60 comp rx=0 tx=0).stop 2023-07-14T19:24:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 shutdown_connections 2023-07-14T19:24:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f35600a1240 0x7f35600a36f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3570100690 0x7f357019f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35700feb10 0x7f3570197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 --2- 172.21.15.53:0/2555602464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35700ffd00 0x7f35701981c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 >> 172.21.15.53:0/2555602464 conn(0x7f35700fa130 msgr2=0x7f35701031c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 shutdown_connections 2023-07-14T19:24:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.592+0000 7f3576fbc700 1 -- 172.21.15.53:0/2555602464 wait complete. 2023-07-14T19:24:53.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:24:53.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:53 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3725529666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:53 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3079288879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:53 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/4085621114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:53.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:53 smithi070 ceph-mon[98015]: pgmap v288: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 63 B/s, 1 objects/s recovering 2023-07-14T19:24:53.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.734+0000 7fdd001f4700 1 -- 172.21.15.53:0/3125922332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 msgr2=0x7fdcf8101380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.734+0000 7fdd001f4700 1 --2- 172.21.15.53:0/3125922332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 0x7fdcf8101380 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fdce8009990 tx=0x7fdce8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.739 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.734+0000 7fdd001f4700 1 -- 172.21.15.53:0/3125922332 shutdown_connections 2023-07-14T19:24:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.734+0000 7fdd001f4700 1 --2- 172.21.15.53:0/3125922332 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf81018c0 0x7fdcf8108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.734+0000 7fdd001f4700 1 --2- 172.21.15.53:0/3125922332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcf80fdec0 0x7fdcf80fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.734+0000 7fdd001f4700 1 --2- 172.21.15.53:0/3125922332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 0x7fdcf8101380 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.734+0000 7fdd001f4700 1 -- 172.21.15.53:0/3125922332 >> 172.21.15.53:0/3125922332 conn(0x7fdcf80f98a0 msgr2=0x7fdcf80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.734+0000 7fdd001f4700 1 -- 172.21.15.53:0/3125922332 shutdown_connections 2023-07-14T19:24:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.734+0000 7fdd001f4700 1 -- 172.21.15.53:0/3125922332 wait complete. 2023-07-14T19:24:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdd001f4700 1 Processor -- start 2023-07-14T19:24:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdd001f4700 1 -- start start 2023-07-14T19:24:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdd001f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80fdec0 0x7fdcf81958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdd001f4700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 0x7fdcf8195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdd001f4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcf81018c0 0x7fdcf819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdd001f4700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf819d470 con 0x7fdcf8100f30 2023-07-14T19:24:53.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdd001f4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf819d5b0 con 0x7fdcf81018c0 2023-07-14T19:24:53.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdd001f4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf819d740 con 0x7fdcf80fdec0 2023-07-14T19:24:53.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfd78f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 0x7fdcf8195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfd78f700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 0x7fdcf8195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51340/0 (socket says 172.21.15.53:51340) 2023-07-14T19:24:53.743 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfd78f700 1 -- 172.21.15.53:0/1155931113 learned_addr learned my addr 172.21.15.53:0/1155931113 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:53.744 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfe791700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcf81018c0 0x7fdcf819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.744 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfdf90700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80fdec0 0x7fdcf81958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.744 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfd78f700 1 -- 172.21.15.53:0/1155931113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80fdec0 msgr2=0x7fdcf81958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.744 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfd78f700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80fdec0 0x7fdcf81958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.744 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfd78f700 1 -- 172.21.15.53:0/1155931113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcf81018c0 msgr2=0x7fdcf819cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.744 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfd78f700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcf81018c0 0x7fdcf819cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.735+0000 7fdcfd78f700 1 -- 172.21.15.53:0/1155931113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce80095f0 con 0x7fdcf8100f30 2023-07-14T19:24:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.736+0000 7fdcfd78f700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 0x7fdcf8195e30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdce8005f50 tx=0x7fdce80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.736+0000 7fdceeffd700 1 -- 172.21.15.53:0/1155931113 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce801d070 con 0x7fdcf8100f30 2023-07-14T19:24:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.736+0000 7fdceeffd700 1 -- 172.21.15.53:0/1155931113 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdce8021410 con 0x7fdcf8100f30 2023-07-14T19:24:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.736+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf819d9c0 con 0x7fdcf8100f30 2023-07-14T19:24:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.736+0000 7fdceeffd700 1 -- 172.21.15.53:0/1155931113 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce8017650 con 0x7fdcf8100f30 2023-07-14T19:24:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.736+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf819dec0 con 0x7fdcf8100f30 2023-07-14T19:24:53.746 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.737+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf81072d0 con 0x7fdcf8100f30 2023-07-14T19:24:53.746 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.738+0000 7fdceeffd700 1 -- 172.21.15.53:0/1155931113 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fdce8021580 con 0x7fdcf8100f30 2023-07-14T19:24:53.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.739+0000 7fdceeffd700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdce40a1260 0x7fdce40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:53.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.739+0000 7fdceeffd700 1 -- 172.21.15.53:0/1155931113 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fdce80ce480 con 0x7fdcf8100f30 2023-07-14T19:24:53.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.739+0000 7fdcfdf90700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdce40a1260 0x7fdce40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:53.747 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.739+0000 7fdcfdf90700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdce40a1260 0x7fdce40a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8067500 tx=0x7fdcf4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:53.748 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.742+0000 7fdceeffd700 1 -- 172.21.15.53:0/1155931113 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdce8097b30 con 0x7fdcf8100f30 2023-07-14T19:24:53.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:53 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3725529666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:53.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:53 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3079288879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:53.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:53 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/4085621114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:24:53.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:53 smithi104 ceph-mon[98435]: pgmap v288: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 63 B/s, 1 objects/s recovering 2023-07-14T19:24:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.942+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdcf802ca70 con 0x7fdcf8100f30 2023-07-14T19:24:53.944 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.942+0000 7fdceeffd700 1 -- 172.21.15.53:0/1155931113 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdce809b150 con 0x7fdcf8100f30 2023-07-14T19:24:53.944 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2023-07-14T19:24:53.945 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdce40a1260 msgr2=0x7fdce40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdce40a1260 0x7fdce40a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8067500 tx=0x7fdcf4009260 comp rx=0 tx=0).stop 2023-07-14T19:24:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 msgr2=0x7fdcf8195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 0x7fdcf8195e30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdce8005f50 tx=0x7fdce80057b0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 shutdown_connections 2023-07-14T19:24:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fdce40a1260 0x7fdce40a3710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcf81018c0 0x7fdcf819cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdcf80fdec0 0x7fdcf81958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 --2- 172.21.15.53:0/1155931113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcf8100f30 0x7fdcf8195e30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 >> 172.21.15.53:0/1155931113 conn(0x7fdcf80f98a0 msgr2=0x7fdcf8104410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 shutdown_connections 2023-07-14T19:24:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:53.944+0000 7fdd001f4700 1 -- 172.21.15.53:0/1155931113 wait complete. 2023-07-14T19:24:53.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:24:54.084 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 -- 172.21.15.53:0/1309140483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800feb30 msgr2=0x7f7a800fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 --2- 172.21.15.53:0/1309140483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800feb30 0x7f7a800fef40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7a74009990 tx=0x7f7a74009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 -- 172.21.15.53:0/1309140483 shutdown_connections 2023-07-14T19:24:54.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 --2- 172.21.15.53:0/1309140483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a801006b0 0x7f7a80106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 --2- 172.21.15.53:0/1309140483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800feb30 0x7f7a800fef40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 --2- 172.21.15.53:0/1309140483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a800ffd20 0x7f7a80100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 -- 172.21.15.53:0/1309140483 >> 172.21.15.53:0/1309140483 conn(0x7f7a800fa110 msgr2=0x7f7a800fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.085 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 -- 172.21.15.53:0/1309140483 shutdown_connections 2023-07-14T19:24:54.086 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 -- 172.21.15.53:0/1309140483 wait complete. 2023-07-14T19:24:54.086 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 Processor -- start 2023-07-14T19:24:54.086 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.083+0000 7f7a88336700 1 -- start start 2023-07-14T19:24:54.087 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a88336700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a800feb30 0x7f7a801948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a88336700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800ffd20 0x7f7a80194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a88336700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a801006b0 0x7f7a8019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a88336700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8019c430 con 0x7f7a800ffd20 2023-07-14T19:24:54.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a88336700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8019c570 con 0x7f7a800feb30 2023-07-14T19:24:54.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a88336700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8019c700 con 0x7f7a801006b0 2023-07-14T19:24:54.088 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a858d1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800ffd20 0x7f7a80194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.089 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a858d1700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800ffd20 0x7f7a80194df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51354/0 (socket says 172.21.15.53:51354) 2023-07-14T19:24:54.089 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a858d1700 1 -- 172.21.15.53:0/261341442 learned_addr learned my addr 172.21.15.53:0/261341442 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:54.089 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a860d2700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a800feb30 0x7f7a801948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.089 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a868d3700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a801006b0 0x7f7a8019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.089 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a858d1700 1 -- 172.21.15.53:0/261341442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a801006b0 msgr2=0x7f7a8019be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.090 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a858d1700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a801006b0 0x7f7a8019be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.090 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a858d1700 1 -- 172.21.15.53:0/261341442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a800feb30 msgr2=0x7f7a801948b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.090 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a858d1700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a800feb30 0x7f7a801948b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.090 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a858d1700 1 -- 172.21.15.53:0/261341442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a740095f0 con 0x7f7a800ffd20 2023-07-14T19:24:54.090 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a858d1700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800ffd20 0x7f7a80194df0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c920 tx=0x7f7a7c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.090 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a737fe700 1 -- 172.21.15.53:0/261341442 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c007b00 con 0x7f7a800ffd20 2023-07-14T19:24:54.090 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a737fe700 1 -- 172.21.15.53:0/261341442 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a7c00ce70 con 0x7f7a800ffd20 2023-07-14T19:24:54.090 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8019c9e0 con 0x7f7a800ffd20 2023-07-14T19:24:54.091 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a737fe700 1 -- 172.21.15.53:0/261341442 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c0054e0 con 0x7f7a800ffd20 2023-07-14T19:24:54.091 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.084+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8019cf40 con 0x7f7a800ffd20 2023-07-14T19:24:54.091 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.085+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a801060d0 con 0x7f7a800ffd20 2023-07-14T19:24:54.091 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.086+0000 7f7a737fe700 1 -- 172.21.15.53:0/261341442 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7a7c004750 con 0x7f7a800ffd20 2023-07-14T19:24:54.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.087+0000 7f7a737fe700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7a6c0a12a0 0x7f7a6c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.087+0000 7f7a860d2700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7a6c0a12a0 0x7f7a6c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.087+0000 7f7a737fe700 1 -- 172.21.15.53:0/261341442 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f7a7c0cc980 con 0x7f7a800ffd20 2023-07-14T19:24:54.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.088+0000 7f7a860d2700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7a6c0a12a0 0x7f7a6c0a3750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400b5c0 tx=0x7f7a74005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.092 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.091+0000 7f7a737fe700 1 -- 172.21.15.53:0/261341442 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7a7c096020 con 0x7f7a800ffd20 2023-07-14T19:24:54.336 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.335+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a8002c8e0 con 0x7f7a800ffd20 2023-07-14T19:24:54.337 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.336+0000 7f7a737fe700 1 -- 172.21.15.53:0/261341442 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f7a7c01d070 con 0x7f7a800ffd20 2023-07-14T19:24:54.338 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:54.338 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:23:37.894565+0000","modified":"2023-07-14T19:23:43.289341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24491,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24484":{"gid":24484,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24491":{"gid":24491,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6839/2496341573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":2496341573},{"type":"v1","addr":"172.21.15.70:6839","nonce":2496341573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:24:54.340 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7a6c0a12a0 msgr2=0x7f7a6c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.340 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7a6c0a12a0 0x7f7a6c0a3750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400b5c0 tx=0x7f7a74005960 comp rx=0 tx=0).stop 2023-07-14T19:24:54.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800ffd20 msgr2=0x7f7a80194df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800ffd20 0x7f7a80194df0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c920 tx=0x7f7a7c00cc30 comp rx=0 tx=0).stop 2023-07-14T19:24:54.341 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 shutdown_connections 2023-07-14T19:24:54.342 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f7a6c0a12a0 0x7f7a6c0a3750 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.342 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a801006b0 0x7f7a8019be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.342 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a800feb30 0x7f7a801948b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.342 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 --2- 172.21.15.53:0/261341442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a800ffd20 0x7f7a80194df0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.343 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 >> 172.21.15.53:0/261341442 conn(0x7f7a800fa110 msgr2=0x7f7a80103210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.343 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 shutdown_connections 2023-07-14T19:24:54.343 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.338+0000 7f7a88336700 1 -- 172.21.15.53:0/261341442 wait complete. 2023-07-14T19:24:54.343 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 23 2023-07-14T19:24:54.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:24:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.476+0000 7f3986398700 1 -- 172.21.15.53:0/3579365521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3980105920 msgr2=0x7f3980107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.476+0000 7f3986398700 1 --2- 172.21.15.53:0/3579365521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3980105920 0x7f3980107d20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f397800b0a0 tx=0x7f397800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.476+0000 7f3986398700 1 -- 172.21.15.53:0/3579365521 shutdown_connections 2023-07-14T19:24:54.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.476+0000 7f3986398700 1 --2- 172.21.15.53:0/3579365521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3980105920 0x7f3980107d20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.476+0000 7f3986398700 1 --2- 172.21.15.53:0/3579365521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39801006e0 0x7f3980102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.476+0000 7f3986398700 1 --2- 172.21.15.53:0/3579365521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3980103000 0x7f39801053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.476+0000 7f3986398700 1 -- 172.21.15.53:0/3579365521 >> 172.21.15.53:0/3579365521 conn(0x7f39800fa100 msgr2=0x7f39800fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 -- 172.21.15.53:0/3579365521 shutdown_connections 2023-07-14T19:24:54.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 -- 172.21.15.53:0/3579365521 wait complete. 2023-07-14T19:24:54.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 Processor -- start 2023-07-14T19:24:54.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 -- start start 2023-07-14T19:24:54.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39801006e0 0x7f3980100050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3980103000 0x7f39800fe690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3980105920 0x7f39800febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800ff370 con 0x7f39801006e0 2023-07-14T19:24:54.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800ff4e0 con 0x7f3980105920 2023-07-14T19:24:54.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.477+0000 7f3986398700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398019fa20 con 0x7f3980103000 2023-07-14T19:24:54.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39801006e0 0x7f3980100050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39801006e0 0x7f3980100050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51366/0 (socket says 172.21.15.53:51366) 2023-07-14T19:24:54.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397ffff700 1 -- 172.21.15.53:0/1415863495 learned_addr learned my addr 172.21.15.53:0/1415863495 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:54.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397f7fe700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3980103000 0x7f39800fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f3984935700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3980105920 0x7f39800febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397ffff700 1 -- 172.21.15.53:0/1415863495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3980103000 msgr2=0x7f39800fe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.488 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397ffff700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3980103000 0x7f39800fe690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397ffff700 1 -- 172.21.15.53:0/1415863495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3980105920 msgr2=0x7f39800febd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397ffff700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3980105920 0x7f39800febd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397ffff700 1 -- 172.21.15.53:0/1415863495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3978009d00 con 0x7f39801006e0 2023-07-14T19:24:54.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397ffff700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39801006e0 0x7f3980100050 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f397000e9a0 tx=0x7f397000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397d7fa700 1 -- 172.21.15.53:0/1415863495 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970009790 con 0x7f39801006e0 2023-07-14T19:24:54.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397d7fa700 1 -- 172.21.15.53:0/1415863495 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3970010430 con 0x7f39801006e0 2023-07-14T19:24:54.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398019fd00 con 0x7f39801006e0 2023-07-14T19:24:54.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f397d7fa700 1 -- 172.21.15.53:0/1415863495 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3970018680 con 0x7f39801006e0 2023-07-14T19:24:54.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.478+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39801a01b0 con 0x7f39801006e0 2023-07-14T19:24:54.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.479+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39800fbc70 con 0x7f39801006e0 2023-07-14T19:24:54.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.480+0000 7f397d7fa700 1 -- 172.21.15.53:0/1415863495 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f39700105a0 con 0x7f39801006e0 2023-07-14T19:24:54.492 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.481+0000 7f397d7fa700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f39680a12a0 0x7f39680a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.493 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.481+0000 7f397f7fe700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f39680a12a0 0x7f39680a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.493 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.481+0000 7f397d7fa700 1 -- 172.21.15.53:0/1415863495 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f3970014070 con 0x7f39801006e0 2023-07-14T19:24:54.493 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.482+0000 7f397f7fe700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f39680a12a0 0x7f39680a3750 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3974009c60 tx=0x7f3974009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.493 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.486+0000 7f397d7fa700 1 -- 172.21.15.53:0/1415863495 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3970051780 con 0x7f39801006e0 2023-07-14T19:24:54.688 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.687+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f398002cb20 con 0x7f39801006e0 2023-07-14T19:24:54.690 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.688+0000 7f397d7fa700 1 -- 172.21.15.53:0/1415863495 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22459 (secure 0 0 0) 0x7f3970096200 con 0x7f39801006e0 2023-07-14T19:24:54.691 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:24:54.691 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:24:42.310475+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:24:54.693 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f39680a12a0 msgr2=0x7f39680a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.693 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f39680a12a0 0x7f39680a3750 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3974009c60 tx=0x7f3974009260 comp rx=0 tx=0).stop 2023-07-14T19:24:54.693 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39801006e0 msgr2=0x7f3980100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.694 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39801006e0 0x7f3980100050 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f397000e9a0 tx=0x7f397000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.694 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 shutdown_connections 2023-07-14T19:24:54.694 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f39680a12a0 0x7f39680a3750 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.694 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3980105920 0x7f39800febd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.695 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39801006e0 0x7f3980100050 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.695 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 --2- 172.21.15.53:0/1415863495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3980103000 0x7f39800fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.695 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 >> 172.21.15.53:0/1415863495 conn(0x7f39800fa100 msgr2=0x7f3980101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.695 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 shutdown_connections 2023-07-14T19:24:54.696 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.691+0000 7f3986398700 1 -- 172.21.15.53:0/1415863495 wait complete. 2023-07-14T19:24:54.704 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:54 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2555602464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:54.704 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:54 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1155931113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:54.704 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:54 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/261341442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:54.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T19:23:37.894565+0000', 'modified': '2023-07-14T19:23:43.289341+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24527, 'mds_1': 24491, 'mds_2': 24461}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6835/1013468175', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6834', 'nonce': 1013468175}, {'type': 'v1', 'addr': '172.21.15.104:6835', 'nonce': 1013468175}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24461': {'gid': 24461, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.70:6835/2384359054', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6834', 'nonce': 2384359054}, {'type': 'v1', 'addr': '172.21.15.70:6835', 'nonce': 2384359054}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6837/1255142015', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6836', 'nonce': 1255142015}, {'type': 'v1', 'addr': '172.21.15.104:6837', 'nonce': 1255142015}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6839/1650908884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 1650908884}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 1650908884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24491': {'gid': 24491, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.70:6837/3730963239', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 3730963239}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3730963239}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6839/2496341573', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 2496341573}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 2496341573}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T19:24:54.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-14T19:24:54.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-07-14T19:24:54.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:54 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2555602464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:54.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:54 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1155931113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:54.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:54 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/261341442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:54.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:54 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2555602464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:24:54.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:54 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1155931113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:24:54.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:54 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/261341442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:24:54.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 -- 172.21.15.53:0/1662149877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 msgr2=0x7f3168106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.833 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 --2- 172.21.15.53:0/1662149877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 0x7f3168106f50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f316000b0a0 tx=0x7f316000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 -- 172.21.15.53:0/1662149877 shutdown_connections 2023-07-14T19:24:54.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 --2- 172.21.15.53:0/1662149877 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 0x7f3168106f50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 --2- 172.21.15.53:0/1662149877 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31680feb10 0x7f31680fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 --2- 172.21.15.53:0/1662149877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31680ffd00 0x7f3168100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 -- 172.21.15.53:0/1662149877 >> 172.21.15.53:0/1662149877 conn(0x7f31680fa130 msgr2=0x7f31680fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 -- 172.21.15.53:0/1662149877 shutdown_connections 2023-07-14T19:24:54.834 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 -- 172.21.15.53:0/1662149877 wait complete. 2023-07-14T19:24:54.835 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.832+0000 7f316da1b700 1 Processor -- start 2023-07-14T19:24:54.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f316da1b700 1 -- start start 2023-07-14T19:24:54.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f316da1b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31680feb10 0x7f3168102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.836 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f316da1b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31680ffd00 0x7f31681010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f316da1b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 0x7f3168101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f316da1b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3168101dc0 con 0x7f3168100690 2023-07-14T19:24:54.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f316da1b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3168101f30 con 0x7f31680feb10 2023-07-14T19:24:54.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f316da1b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316819faa0 con 0x7f31680ffd00 2023-07-14T19:24:54.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31677fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 0x7f3168101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31677fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 0x7f3168101620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51384/0 (socket says 172.21.15.53:51384) 2023-07-14T19:24:54.837 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31677fe700 1 -- 172.21.15.53:0/3942370248 learned_addr learned my addr 172.21.15.53:0/3942370248 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:54.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f3166ffd700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31680feb10 0x7f3168102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31667fc700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31680ffd00 0x7f31681010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31677fe700 1 -- 172.21.15.53:0/3942370248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31680ffd00 msgr2=0x7f31681010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31677fe700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31680ffd00 0x7f31681010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31677fe700 1 -- 172.21.15.53:0/3942370248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31680feb10 msgr2=0x7f3168102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31677fe700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31680feb10 0x7f3168102aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.838 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31677fe700 1 -- 172.21.15.53:0/3942370248 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158009520 con 0x7f3168100690 2023-07-14T19:24:54.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31667fc700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31680ffd00 0x7f31681010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:54.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.833+0000 7f31677fe700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 0x7f3168101620 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f316000b780 tx=0x7f316000bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.834+0000 7f316ca19700 1 -- 172.21.15.53:0/3942370248 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316000e040 con 0x7f3168100690 2023-07-14T19:24:54.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.834+0000 7f316ca19700 1 -- 172.21.15.53:0/3942370248 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3160004c60 con 0x7f3168100690 2023-07-14T19:24:54.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.834+0000 7f316da1b700 1 -- 172.21.15.53:0/3942370248 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3160009d00 con 0x7f3168100690 2023-07-14T19:24:54.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.834+0000 7f316ca19700 1 -- 172.21.15.53:0/3942370248 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316001b6b0 con 0x7f3168100690 2023-07-14T19:24:54.839 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.834+0000 7f316da1b700 1 -- 172.21.15.53:0/3942370248 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31681a0010 con 0x7f3168100690 2023-07-14T19:24:54.845 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.836+0000 7f316ca19700 1 -- 172.21.15.53:0/3942370248 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f316001b810 con 0x7f3168100690 2023-07-14T19:24:54.845 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.837+0000 7f316ca19700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31500a11e0 0x7f31500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.845 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.837+0000 7f3166ffd700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31500a11e0 0x7f31500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.845 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.837+0000 7f316ca19700 1 -- 172.21.15.53:0/3942370248 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f31600ce310 con 0x7f3168100690 2023-07-14T19:24:54.845 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.837+0000 7f3166ffd700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31500a11e0 0x7f31500a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f31680ffb60 tx=0x7f315c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3179bb3b80 1 Processor -- start 2023-07-14T19:24:54.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3179bb3b80 1 -- start start 2023-07-14T19:24:54.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddb8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 0x556dcddb8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddb98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcdcbcc60 con 0x556dcddb9460 2023-07-14T19:24:54.846 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcdd0f9f0 con 0x556dcddb7f10 2023-07-14T19:24:54.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcdd08e80 con 0x556dcddb8860 2023-07-14T19:24:54.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31677fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddb98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31677fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddb98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51392/0 (socket says 172.21.15.53:51392) 2023-07-14T19:24:54.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31677fe700 1 -- 172.21.15.53:0/3302654294 learned_addr learned my addr 172.21.15.53:0/3302654294 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:54.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3166ffd700 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddb8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31667fc700 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 0x556dcddb8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31677fe700 1 -- 172.21.15.53:0/3302654294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 msgr2=0x556dcddb8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:54.847 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31677fe700 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 0x556dcddb8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.848 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31677fe700 1 -- 172.21.15.53:0/3302654294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 msgr2=0x556dcddb8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:54.848 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31677fe700 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddb8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.848 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31677fe700 1 -- 172.21.15.53:0/3302654294 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556dcdb36d90 con 0x556dcddb9460 2023-07-14T19:24:54.848 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f3166ffd700 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddb8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:54.848 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.838+0000 7f31677fe700 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddb98d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3160097b80 tx=0x7f316009b2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a820b5bbd6763458 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.848 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f314dffb700 1 -- 172.21.15.53:0/3302654294 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316009f6b0 con 0x556dcddb9460 2023-07-14T19:24:54.848 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f314dffb700 1 -- 172.21.15.53:0/3302654294 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31600a3a40 con 0x556dcddb9460 2023-07-14T19:24:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f314dffb700 1 -- 172.21.15.53:0/3302654294 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316009ab70 con 0x556dcddb9460 2023-07-14T19:24:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3302654294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 msgr2=0x556dcddb98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddb98d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3160097b80 tx=0x7f316009b2b0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3302654294 shutdown_connections 2023-07-14T19:24:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddb98d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddb8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3302654294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 0x556dcddb8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3302654294 >> 172.21.15.53:0/3302654294 conn(0x556dcddad630 msgr2=0x556dcddb2980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3302654294 shutdown_connections 2023-07-14T19:24:54.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.839+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3302654294 wait complete. 2023-07-14T19:24:54.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3179bb3b80 1 -- start start 2023-07-14T19:24:54.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3179bb3b80 1 Processor -- start 2023-07-14T19:24:54.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddc16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 0x556dcddc1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.850 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddc8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcdca8a70 con 0x556dcddb9460 2023-07-14T19:24:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcdca8770 con 0x556dcddb7f10 2023-07-14T19:24:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcdb8c8d0 con 0x556dcddb8860 2023-07-14T19:24:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3179bb3b80 10 client.0 _ll_register_callbacks cb 0x556dcdca6710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f31677fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddc8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f31677fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddc8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51398/0 (socket says 172.21.15.53:51398) 2023-07-14T19:24:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f31677fe700 1 -- 172.21.15.53:0/1033752489 learned_addr learned my addr 172.21.15.53:0/1033752489 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f31667fc700 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 0x556dcddc1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.840+0000 7f3166ffd700 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddc16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f31677fe700 1 -- 172.21.15.53:0/1033752489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 msgr2=0x556dcddc1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f31677fe700 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 0x556dcddc1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f31677fe700 1 -- 172.21.15.53:0/1033752489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 msgr2=0x556dcddc16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f31677fe700 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddc16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f31677fe700 1 -- 172.21.15.53:0/1033752489 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31681a0010 con 0x556dcddb9460 2023-07-14T19:24:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3166ffd700 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddc16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f31677fe700 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddc8c60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f31600a3510 tx=0x7f31600bb1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da039446d9eeaee server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3146ffd700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3146ffd700 1 -- 172.21.15.53:0/1033752489 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316009ad70 con 0x556dcddb9460 2023-07-14T19:24:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3146ffd700 1 -- 172.21.15.53:0/1033752489 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f31600b7030 con 0x556dcddb9460 2023-07-14T19:24:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1033752489 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556dcdc624a0 con 0x556dcddb9460 2023-07-14T19:24:54.854 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3146ffd700 1 -- 172.21.15.53:0/1033752489 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31600a35b0 con 0x556dcddb9460 2023-07-14T19:24:54.854 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3179bb3b80 10 client.15198 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:54.854 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1033752489 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556dcdb1fc10 con 0x556dcddb9460 2023-07-14T19:24:54.854 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3146ffd700 1 -- 172.21.15.53:0/1033752489 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556dcdc624a0 con 0x556dcddb9460 2023-07-14T19:24:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.841+0000 7f3146ffd700 1 -- 172.21.15.53:0/1033752489 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f316009cae0 con 0x556dcddb9460 2023-07-14T19:24:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.842+0000 7f3179bb3b80 10 client.15198 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.842+0000 7f3179bb3b80 10 client.15198 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T19:24:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.842+0000 7f3179bb3b80 20 client.15198 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.842+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x556dcddcc370 0x556dcddcbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.842+0000 7f3179bb3b80 4 client.15198 mds_command: new command op to 24461 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:24:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.842+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1033752489 --> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556dcdd86d50 con 0x556dcddcc370 2023-07-14T19:24:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.842+0000 7f3166ffd700 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x556dcddcc370 0x556dcddcbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.856 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.842+0000 7f3166ffd700 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x556dcddcc370 0x556dcddcbb00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4295d9c1ede98ac server_cookie=4dad07b57ef7713d in_seq=0 out_seq=0 2023-07-14T19:24:54.856 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.842+0000 7f3146ffd700 10 client.15198 ms_handle_connect on v2:172.21.15.70:6834/2384359054 2023-07-14T19:24:54.856 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3146ffd700 1 -- 172.21.15.53:0/1033752489 <== mds.2 v2:172.21.15.70:6834/2384359054 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x556dcdd86d50 con 0x556dcddcc370 2023-07-14T19:24:54.856 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3146ffd700 10 client.15198 handle_command_reply: tid=0 2023-07-14T19:24:54.856 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 client.15198 shutdown 2023-07-14T19:24:54.856 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1033752489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 msgr2=0x556dcddc8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddc8c60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f31600a3510 tx=0x7f31600bb1e0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1033752489 shutdown_connections 2023-07-14T19:24:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x556dcddcc370 0x556dcddcbb00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddc8c60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb7f10 0x556dcddc16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1033752489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb8860 0x556dcddc1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1033752489 >> 172.21.15.53:0/1033752489 conn(0x556dcddad630 msgr2=0x556dcddb1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3146ffd700 0 client.15198 ms_handle_reset on v2:172.21.15.70:6834/2384359054 2023-07-14T19:24:54.858 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1033752489 shutdown_connections 2023-07-14T19:24:54.858 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.843+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1033752489 wait complete. 2023-07-14T19:24:54.858 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.844+0000 7f3179bb3b80 20 client.15198 trim_cache size 0 max 16384 2023-07-14T19:24:54.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f3179bb3b80 1 Processor -- start 2023-07-14T19:24:54.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f3179bb3b80 1 -- start start 2023-07-14T19:24:54.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb7f10 0x556dcddbf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 0x556dcddd50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddbc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcddc25e0 con 0x556dcddb9460 2023-07-14T19:24:54.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcdd86d50 con 0x556dcddb8860 2023-07-14T19:24:54.870 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcdc624a0 con 0x556dcddb7f10 2023-07-14T19:24:54.871 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f31677fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddbc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.871 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f31677fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddbc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51406/0 (socket says 172.21.15.53:51406) 2023-07-14T19:24:54.871 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.861+0000 7f31677fe700 1 -- 172.21.15.53:0/1595013273 learned_addr learned my addr 172.21.15.53:0/1595013273 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:54.871 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f31667fc700 1 --2- 172.21.15.53:0/1595013273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 0x556dcddd50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.871 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f3166ffd700 1 --2- 172.21.15.53:0/1595013273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb7f10 0x556dcddbf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f31677fe700 1 -- 172.21.15.53:0/1595013273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb7f10 msgr2=0x556dcddbf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:54.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f31677fe700 1 --2- 172.21.15.53:0/1595013273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb7f10 0x556dcddbf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f31677fe700 1 -- 172.21.15.53:0/1595013273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 msgr2=0x556dcddd50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:54.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f31677fe700 1 --2- 172.21.15.53:0/1595013273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 0x556dcddd50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f31677fe700 1 -- 172.21.15.53:0/1595013273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556dcdb5d200 con 0x556dcddb9460 2023-07-14T19:24:54.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f31677fe700 1 --2- 172.21.15.53:0/1595013273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddbc380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f316009b840 tx=0x7f3160097af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8b473a45b647d0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f314e7fc700 1 -- 172.21.15.53:0/1595013273 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316009b1c0 con 0x556dcddb9460 2023-07-14T19:24:54.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f314e7fc700 1 -- 172.21.15.53:0/1595013273 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f316009cae0 con 0x556dcddb9460 2023-07-14T19:24:54.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.862+0000 7f314e7fc700 1 -- 172.21.15.53:0/1595013273 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31600860f0 con 0x556dcddb9460 2023-07-14T19:24:54.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1595013273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 msgr2=0x556dcddbc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1595013273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddbc380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f316009b840 tx=0x7f3160097af0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1595013273 shutdown_connections 2023-07-14T19:24:54.873 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1595013273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb9460 0x556dcddbc380 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1595013273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb7f10 0x556dcddbf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/1595013273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 0x556dcddd50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1595013273 >> 172.21.15.53:0/1595013273 conn(0x556dcddad630 msgr2=0x556dcddc03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1595013273 shutdown_connections 2023-07-14T19:24:54.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 -- 172.21.15.53:0/1595013273 wait complete. 2023-07-14T19:24:54.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 -- start start 2023-07-14T19:24:54.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 Processor -- start 2023-07-14T19:24:54.874 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb7f10 0x556dcddd0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 0x556dcddd0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb9460 0x556dcddd3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcddd3bd0 con 0x556dcddb7f10 2023-07-14T19:24:54.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcddd3d40 con 0x556dcddb8860 2023-07-14T19:24:54.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.863+0000 7f3179bb3b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dcddd3eb0 con 0x556dcddb9460 2023-07-14T19:24:54.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3179bb3b80 10 client.0 _ll_register_callbacks cb 0x556dcdca6710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:54.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3166ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb7f10 0x556dcddd0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.875 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3166ffd700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb7f10 0x556dcddd0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51408/0 (socket says 172.21.15.53:51408) 2023-07-14T19:24:54.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3166ffd700 1 -- 172.21.15.53:0/3782324804 learned_addr learned my addr 172.21.15.53:0/3782324804 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:54.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f31667fc700 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 0x556dcddd0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f31677fe700 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb9460 0x556dcddd3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3166ffd700 1 -- 172.21.15.53:0/3782324804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb9460 msgr2=0x556dcddd3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:54.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3166ffd700 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb9460 0x556dcddd3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3166ffd700 1 -- 172.21.15.53:0/3782324804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 msgr2=0x556dcddd0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:54.876 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3166ffd700 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 0x556dcddd0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3166ffd700 1 -- 172.21.15.53:0/3782324804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556dcdb1fc10 con 0x556dcddb7f10 2023-07-14T19:24:54.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3166ffd700 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb7f10 0x556dcddd0310 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x556dcddc2740 tx=0x7f315c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbd3bfbe268b06c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:54.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3146ffd700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:54.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3146ffd700 1 -- 172.21.15.53:0/3782324804 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c022040 con 0x556dcddb7f10 2023-07-14T19:24:54.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3146ffd700 1 -- 172.21.15.53:0/3782324804 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f315c01c600 con 0x556dcddb7f10 2023-07-14T19:24:54.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3782324804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f31600a3a40 con 0x556dcddb7f10 2023-07-14T19:24:54.877 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3146ffd700 1 -- 172.21.15.53:0/3782324804 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f315c004020 con 0x556dcddb7f10 2023-07-14T19:24:54.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3179bb3b80 10 client.15204 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:54.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.864+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3782324804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556dcdb35260 con 0x556dcddb7f10 2023-07-14T19:24:54.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f3146ffd700 1 -- 172.21.15.53:0/3782324804 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31600a3a40 con 0x556dcddb7f10 2023-07-14T19:24:54.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f3146ffd700 1 -- 172.21.15.53:0/3782324804 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f315c003680 con 0x556dcddb7f10 2023-07-14T19:24:54.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f3179bb3b80 10 client.15204 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:54.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f3179bb3b80 10 client.15204 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T19:24:54.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f3179bb3b80 20 client.15204 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:54.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x556dcde255f0 0x556dcde21fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:54.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f3179bb3b80 4 client.15204 mds_command: new command op to 24461 tid=0[{"prefix": "status"}] 2023-07-14T19:24:54.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3782324804 --> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556dcdd01e60 con 0x556dcde255f0 2023-07-14T19:24:54.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f31667fc700 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x556dcde255f0 0x556dcde21fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:54.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f3146ffd700 10 client.15204 ms_handle_connect on v2:172.21.15.70:6834/2384359054 2023-07-14T19:24:54.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.865+0000 7f31667fc700 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x556dcde255f0 0x556dcde21fc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5ea46e850017691f server_cookie=27b890996d5904ea in_seq=0 out_seq=0 2023-07-14T19:24:54.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3146ffd700 1 -- 172.21.15.53:0/3782324804 <== mds.2 v2:172.21.15.70:6834/2384359054 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x556dcdd01e60 con 0x556dcde255f0 2023-07-14T19:24:54.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3146ffd700 10 client.15204 handle_command_reply: tid=0 2023-07-14T19:24:54.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 client.15204 shutdown 2023-07-14T19:24:54.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3782324804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb7f10 msgr2=0x556dcddd0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb7f10 0x556dcddd0310 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x556dcddc2740 tx=0x7f315c0047c0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3782324804 shutdown_connections 2023-07-14T19:24:54.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.70:6834/2384359054,v1:172.21.15.70:6835/2384359054] conn(0x556dcde255f0 0x556dcde21fc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.880 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556dcddb9460 0x556dcddd3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556dcddb7f10 0x556dcddd0310 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3146ffd700 0 client.15204 ms_handle_reset on v2:172.21.15.70:6834/2384359054 2023-07-14T19:24:54.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 --2- 172.21.15.53:0/3782324804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556dcddb8860 0x556dcddd0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3782324804 >> 172.21.15.53:0/3782324804 conn(0x556dcddad630 msgr2=0x556dcddc03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3782324804 shutdown_connections 2023-07-14T19:24:54.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 1 -- 172.21.15.53:0/3782324804 wait complete. 2023-07-14T19:24:54.881 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f3179bb3b80 20 client.15204 trim_cache size 0 max 16384 2023-07-14T19:24:54.882 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-07-14T19:24:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:24:54.883 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 2, 2023-07-14T19:24:54.883 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24461, 2023-07-14T19:24:54.883 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2023-07-14T19:24:54.883 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2023-07-14T19:24:54.883 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2023-07-14T19:24:54.883 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 72.581334744000003, 2023-07-14T19:24:54.883 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 23, 2023-07-14T19:24:54.884 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 82, 2023-07-14T19:24:54.884 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 80, 2023-07-14T19:24:54.884 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 98.514007778999996 2023-07-14T19:24:54.884 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-07-14T19:24:54.885 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f316da1b700 1 -- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31500a11e0 msgr2=0x7f31500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.886 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f316da1b700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31500a11e0 0x7f31500a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f31680ffb60 tx=0x7f315c009260 comp rx=0 tx=0).stop 2023-07-14T19:24:54.886 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f316da1b700 1 -- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 msgr2=0x7f3168101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:54.886 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.866+0000 7f316da1b700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 0x7f3168101620 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f316000b780 tx=0x7f316000bd00 comp rx=0 tx=0).stop 2023-07-14T19:24:54.886 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.867+0000 7f316da1b700 1 -- 172.21.15.53:0/3942370248 shutdown_connections 2023-07-14T19:24:54.886 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.867+0000 7f316da1b700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f31500a11e0 0x7f31500a3690 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.886 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.867+0000 7f316da1b700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3168100690 0x7f3168101620 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.886 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.867+0000 7f316da1b700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31680feb10 0x7f3168102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.887 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.867+0000 7f316da1b700 1 --2- 172.21.15.53:0/3942370248 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31680ffd00 0x7f31681010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:54.887 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.867+0000 7f316da1b700 1 -- 172.21.15.53:0/3942370248 >> 172.21.15.53:0/3942370248 conn(0x7f31680fa130 msgr2=0x7f3168067860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:54.887 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.867+0000 7f316da1b700 1 -- 172.21.15.53:0/3942370248 shutdown_connections 2023-07-14T19:24:54.887 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:54.867+0000 7f316da1b700 1 -- 172.21.15.53:0/3942370248 wait complete. 2023-07-14T19:24:54.887 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T19:24:55.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.013+0000 7fa714b48700 1 -- 172.21.15.53:0/1295091250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 msgr2=0x7fa7100fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.013+0000 7fa714b48700 1 --2- 172.21.15.53:0/1295091250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 0x7fa7100fef20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa700009990 tx=0x7fa700009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.013+0000 7fa714b48700 1 -- 172.21.15.53:0/1295091250 shutdown_connections 2023-07-14T19:24:55.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.013+0000 7fa714b48700 1 --2- 172.21.15.53:0/1295091250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa710100690 0x7fa710106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.013+0000 7fa714b48700 1 --2- 172.21.15.53:0/1295091250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 0x7fa7100fef20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.013+0000 7fa714b48700 1 --2- 172.21.15.53:0/1295091250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7100ffd00 0x7fa710100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.013+0000 7fa714b48700 1 -- 172.21.15.53:0/1295091250 >> 172.21.15.53:0/1295091250 conn(0x7fa7100fa130 msgr2=0x7fa7100fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.016 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.013+0000 7fa714b48700 1 -- 172.21.15.53:0/1295091250 shutdown_connections 2023-07-14T19:24:55.016 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.013+0000 7fa714b48700 1 -- 172.21.15.53:0/1295091250 wait complete. 2023-07-14T19:24:55.017 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa714b48700 1 Processor -- start 2023-07-14T19:24:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa714b48700 1 -- start start 2023-07-14T19:24:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa714b48700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 0x7fa7101947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa714b48700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7100ffd00 0x7fa710194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa714b48700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa710100690 0x7fa71019bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa714b48700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71019c370 con 0x7fa7100feb10 2023-07-14T19:24:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa714b48700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71019c4b0 con 0x7fa7100ffd00 2023-07-14T19:24:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa714b48700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71019c640 con 0x7fa710100690 2023-07-14T19:24:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 0x7fa7101947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 0x7fa7101947f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51422/0 (socket says 172.21.15.53:51422) 2023-07-14T19:24:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70e59c700 1 -- 172.21.15.53:0/3542797872 learned_addr learned my addr 172.21.15.53:0/3542797872 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70ed9d700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa710100690 0x7fa71019bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7100ffd00 0x7fa710194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70e59c700 1 -- 172.21.15.53:0/3542797872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa710100690 msgr2=0x7fa71019bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70e59c700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa710100690 0x7fa71019bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70e59c700 1 -- 172.21.15.53:0/3542797872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7100ffd00 msgr2=0x7fa710194d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70e59c700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7100ffd00 0x7fa710194d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.014+0000 7fa70e59c700 1 -- 172.21.15.53:0/3542797872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7000095f0 con 0x7fa7100feb10 2023-07-14T19:24:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.015+0000 7fa70e59c700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 0x7fa7101947f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa7000098e0 tx=0x7fa700005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.015+0000 7fa6ff7fe700 1 -- 172.21.15.53:0/3542797872 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70001f070 con 0x7fa7100feb10 2023-07-14T19:24:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.015+0000 7fa6ff7fe700 1 -- 172.21.15.53:0/3542797872 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa70001adf0 con 0x7fa7100feb10 2023-07-14T19:24:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.015+0000 7fa714b48700 1 -- 172.21.15.53:0/3542797872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7100677a0 con 0x7fa7100feb10 2023-07-14T19:24:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.015+0000 7fa6ff7fe700 1 -- 172.21.15.53:0/3542797872 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7000056e0 con 0x7fa7100feb10 2023-07-14T19:24:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.015+0000 7fa714b48700 1 -- 172.21.15.53:0/3542797872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa710067c50 con 0x7fa7100feb10 2023-07-14T19:24:55.024 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.017+0000 7fa6ff7fe700 1 -- 172.21.15.53:0/3542797872 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa7000047e0 con 0x7fa7100feb10 2023-07-14T19:24:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.018+0000 7fa6ff7fe700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa6f80a11e0 0x7fa6f80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.018+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa6f80a11e0 0x7fa6f80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.018+0000 7fa6ff7fe700 1 -- 172.21.15.53:0/3542797872 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fa7000d0160 con 0x7fa7100feb10 2023-07-14T19:24:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.018+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa6f80a11e0 0x7fa6f80a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa704009bf0 tx=0x7fa704009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa720ce0b80 1 Processor -- start 2023-07-14T19:24:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa720ce0b80 1 -- start start 2023-07-14T19:24:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc4376320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc4376e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc43778d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc427ac60 con 0x560dc4376860 2023-07-14T19:24:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc42cd9f0 con 0x560dc4377460 2023-07-14T19:24:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc42c6e80 con 0x560dc4375f10 2023-07-14T19:24:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa70dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc4376e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa70dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc4376e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51432/0 (socket says 172.21.15.53:51432) 2023-07-14T19:24:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.019+0000 7fa70dd9b700 1 -- 172.21.15.53:0/626425326 learned_addr learned my addr 172.21.15.53:0/626425326 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa70e59c700 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc4376320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa70ed9d700 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc43778d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa70dd9b700 1 -- 172.21.15.53:0/626425326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 msgr2=0x560dc4376320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc4376320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa70dd9b700 1 -- 172.21.15.53:0/626425326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 msgr2=0x560dc43778d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc43778d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa70dd9b700 1 -- 172.21.15.53:0/626425326 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560dc40f4d90 con 0x560dc4376860 2023-07-14T19:24:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa70ed9d700 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc43778d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc4376e80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa704011e40 tx=0x7fa704014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b9a81a94b1733a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa6fcff9700 1 -- 172.21.15.53:0/626425326 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa704004220 con 0x560dc4376860 2023-07-14T19:24:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa6fcff9700 1 -- 172.21.15.53:0/626425326 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa704016070 con 0x560dc4376860 2023-07-14T19:24:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.020+0000 7fa6fcff9700 1 -- 172.21.15.53:0/626425326 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70401f680 con 0x560dc4376860 2023-07-14T19:24:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 -- 172.21.15.53:0/626425326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 msgr2=0x560dc4376e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc4376e80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa704011e40 tx=0x7fa704014000 comp rx=0 tx=0).stop 2023-07-14T19:24:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 -- 172.21.15.53:0/626425326 shutdown_connections 2023-07-14T19:24:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc43778d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc4376320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/626425326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc4376e80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 -- 172.21.15.53:0/626425326 >> 172.21.15.53:0/626425326 conn(0x560dc436b630 msgr2=0x560dc4370980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 -- 172.21.15.53:0/626425326 shutdown_connections 2023-07-14T19:24:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 -- 172.21.15.53:0/626425326 wait complete. 2023-07-14T19:24:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 -- start start 2023-07-14T19:24:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.021+0000 7fa720ce0b80 1 Processor -- start 2023-07-14T19:24:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc437f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc437fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4386ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc4266a70 con 0x560dc4376860 2023-07-14T19:24:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc4266770 con 0x560dc4377460 2023-07-14T19:24:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc414a8d0 con 0x560dc4375f10 2023-07-14T19:24:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa720ce0b80 10 client.0 _ll_register_callbacks cb 0x560dc4264710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc437fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc437fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51438/0 (socket says 172.21.15.53:51438) 2023-07-14T19:24:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70dd9b700 1 -- 172.21.15.53:0/3232676106 learned_addr learned my addr 172.21.15.53:0/3232676106 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70e59c700 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc437f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70ed9d700 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4386ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70dd9b700 1 -- 172.21.15.53:0/3232676106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 msgr2=0x560dc437f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc437f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70dd9b700 1 -- 172.21.15.53:0/3232676106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 msgr2=0x560dc4386ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4386ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70dd9b700 1 -- 172.21.15.53:0/3232676106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa704011b20 con 0x560dc4376860 2023-07-14T19:24:55.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70ed9d700 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4386ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:55.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc437fe60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa7040097d0 tx=0x7fa7040041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b963927d8ad621b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa6f5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:55.032 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/3232676106 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa704005460 con 0x560dc4376860 2023-07-14T19:24:55.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/3232676106 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fa704018070 con 0x560dc4376860 2023-07-14T19:24:55.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.022+0000 7fa720ce0b80 1 -- 172.21.15.53:0/3232676106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560dc42204a0 con 0x560dc4376860 2023-07-14T19:24:55.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/3232676106 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7040055c0 con 0x560dc4376860 2023-07-14T19:24:55.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa720ce0b80 10 client.15216 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:55.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa720ce0b80 1 -- 172.21.15.53:0/3232676106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560dc40ddc10 con 0x560dc4376860 2023-07-14T19:24:55.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/3232676106 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560dc42204a0 con 0x560dc4376860 2023-07-14T19:24:55.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/3232676106 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fa70403d050 con 0x560dc4376860 2023-07-14T19:24:55.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa720ce0b80 10 client.15216 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa720ce0b80 10 client.15216 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:24:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa720ce0b80 20 client.15216 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x560dc438aad0 0x560dc4389eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa720ce0b80 4 client.15216 mds_command: new command op to 24491 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:24:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa720ce0b80 1 -- 172.21.15.53:0/3232676106 --> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560dc4344d50 con 0x560dc438aad0 2023-07-14T19:24:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.023+0000 7fa70e59c700 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x560dc438aad0 0x560dc4389eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.024+0000 7fa70e59c700 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x560dc438aad0 0x560dc4389eb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c9ee5fca59efafa8 server_cookie=853331893327c0c6 in_seq=0 out_seq=0 2023-07-14T19:24:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.024+0000 7fa6f5ffb700 10 client.15216 ms_handle_connect on v2:172.21.15.70:6836/3730963239 2023-07-14T19:24:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.024+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/3232676106 <== mds.1 v2:172.21.15.70:6836/3730963239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x560dc4344d50 con 0x560dc438aad0 2023-07-14T19:24:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.024+0000 7fa6f5ffb700 10 client.15216 handle_command_reply: tid=0 2023-07-14T19:24:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.024+0000 7fa720ce0b80 1 client.15216 shutdown 2023-07-14T19:24:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 -- 172.21.15.53:0/3232676106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 msgr2=0x560dc437fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc437fe60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa7040097d0 tx=0x7fa7040041f0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 -- 172.21.15.53:0/3232676106 shutdown_connections 2023-07-14T19:24:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x560dc438aad0 0x560dc4389eb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4386ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa6f5ffb700 0 client.15216 ms_handle_reset on v2:172.21.15.70:6836/3730963239 2023-07-14T19:24:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc437f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/3232676106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc437fe60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 -- 172.21.15.53:0/3232676106 >> 172.21.15.53:0/3232676106 conn(0x560dc436b630 msgr2=0x560dc436f600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 -- 172.21.15.53:0/3232676106 shutdown_connections 2023-07-14T19:24:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 1 -- 172.21.15.53:0/3232676106 wait complete. 2023-07-14T19:24:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.025+0000 7fa720ce0b80 20 client.15216 trim_cache size 0 max 16384 2023-07-14T19:24:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa720ce0b80 1 Processor -- start 2023-07-14T19:24:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa720ce0b80 1 -- start start 2023-07-14T19:24:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4375f10 0x560dc4389eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4376860 0x560dc438a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4377460 0x560dc4386e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc4371740 con 0x560dc4375f10 2023-07-14T19:24:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc4295790 con 0x560dc4376860 2023-07-14T19:24:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc4156de0 con 0x560dc4377460 2023-07-14T19:24:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa70e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4375f10 0x560dc4389eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa70e59c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4375f10 0x560dc4389eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51444/0 (socket says 172.21.15.53:51444) 2023-07-14T19:24:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.043+0000 7fa70e59c700 1 -- 172.21.15.53:0/1248703352 learned_addr learned my addr 172.21.15.53:0/1248703352 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/1248703352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4376860 0x560dc438a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa70e59c700 1 -- 172.21.15.53:0/1248703352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4377460 msgr2=0x560dc4386e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa70e59c700 1 --2- 172.21.15.53:0/1248703352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4377460 0x560dc4386e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.048 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa70e59c700 1 -- 172.21.15.53:0/1248703352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4376860 msgr2=0x560dc438a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.048 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa70e59c700 1 --2- 172.21.15.53:0/1248703352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4376860 0x560dc438a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.048 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa70e59c700 1 -- 172.21.15.53:0/1248703352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560dc411b200 con 0x560dc4375f10 2023-07-14T19:24:55.048 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/1248703352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4376860 0x560dc438a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:55.048 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa70e59c700 1 --2- 172.21.15.53:0/1248703352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4375f10 0x560dc4389eb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa700000c00 tx=0x7fa7000a3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce58260cc10b8ed server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.048 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa6fd7fa700 1 -- 172.21.15.53:0/1248703352 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7000a5890 con 0x560dc4375f10 2023-07-14T19:24:55.048 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa6fd7fa700 1 -- 172.21.15.53:0/1248703352 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa70009b610 con 0x560dc4375f10 2023-07-14T19:24:55.048 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa6fd7fa700 1 -- 172.21.15.53:0/1248703352 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7000a10b0 con 0x560dc4375f10 2023-07-14T19:24:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa720ce0b80 1 -- 172.21.15.53:0/1248703352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4375f10 msgr2=0x560dc4389eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/1248703352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4375f10 0x560dc4389eb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa700000c00 tx=0x7fa7000a3040 comp rx=0 tx=0).stop 2023-07-14T19:24:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa720ce0b80 1 -- 172.21.15.53:0/1248703352 shutdown_connections 2023-07-14T19:24:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/1248703352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4377460 0x560dc4386e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/1248703352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4375f10 0x560dc4389eb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.044+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/1248703352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4376860 0x560dc438a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 -- 172.21.15.53:0/1248703352 >> 172.21.15.53:0/1248703352 conn(0x560dc436b630 msgr2=0x560dc437e620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 -- 172.21.15.53:0/1248703352 shutdown_connections 2023-07-14T19:24:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 -- 172.21.15.53:0/1248703352 wait complete. 2023-07-14T19:24:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 -- start start 2023-07-14T19:24:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 Processor -- start 2023-07-14T19:24:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc4383090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc43835d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4385ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc43863e0 con 0x560dc4376860 2023-07-14T19:24:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc4386550 con 0x560dc4377460 2023-07-14T19:24:55.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x560dc4386710 con 0x560dc4375f10 2023-07-14T19:24:55.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa720ce0b80 10 client.0 _ll_register_callbacks cb 0x560dc4264710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:55.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa70dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc43835d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa70dd9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc43835d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51452/0 (socket says 172.21.15.53:51452) 2023-07-14T19:24:55.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa70dd9b700 1 -- 172.21.15.53:0/172541842 learned_addr learned my addr 172.21.15.53:0/172541842 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.045+0000 7fa70e59c700 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc4383090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.054 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa70ed9d700 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4385ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa70dd9b700 1 -- 172.21.15.53:0/172541842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 msgr2=0x560dc4383090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc4383090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa70dd9b700 1 -- 172.21.15.53:0/172541842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 msgr2=0x560dc4385ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4385ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa70dd9b700 1 -- 172.21.15.53:0/172541842 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa710067c50 con 0x560dc4376860 2023-07-14T19:24:55.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa70ed9d700 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4385ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:55.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa70dd9b700 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc43835d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa704011770 tx=0x7fa7040094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df96de4dd561a262 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.055 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa6f5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/172541842 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa704005670 con 0x560dc4376860 2023-07-14T19:24:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/172541842 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fa704018040 con 0x560dc4376860 2023-07-14T19:24:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa720ce0b80 1 -- 172.21.15.53:0/172541842 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa704016070 con 0x560dc4376860 2023-07-14T19:24:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/172541842 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa704027720 con 0x560dc4376860 2023-07-14T19:24:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa720ce0b80 10 client.15222 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa720ce0b80 1 -- 172.21.15.53:0/172541842 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x560dc40f3260 con 0x560dc4376860 2023-07-14T19:24:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/172541842 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa704016070 con 0x560dc4376860 2023-07-14T19:24:55.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/172541842 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fa704048050 con 0x560dc4376860 2023-07-14T19:24:55.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa720ce0b80 10 client.15222 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:55.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa720ce0b80 10 client.15222 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:24:55.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa720ce0b80 20 client.15222 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:55.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x560dc43e42a0 0x560dc43e0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa720ce0b80 4 client.15222 mds_command: new command op to 24491 tid=0[{"prefix": "status"}] 2023-07-14T19:24:55.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.046+0000 7fa720ce0b80 1 -- 172.21.15.53:0/172541842 --> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560dc42bfe60 con 0x560dc43e42a0 2023-07-14T19:24:55.057 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa70e59c700 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x560dc43e42a0 0x560dc43e0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa70e59c700 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x560dc43e42a0 0x560dc43e0c30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=faff596038900afe server_cookie=dc46e517f3be7d5 in_seq=0 out_seq=0 2023-07-14T19:24:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa6f5ffb700 10 client.15222 ms_handle_connect on v2:172.21.15.70:6836/3730963239 2023-07-14T19:24:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa6f5ffb700 1 -- 172.21.15.53:0/172541842 <== mds.1 v2:172.21.15.70:6836/3730963239 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x560dc42bfe60 con 0x560dc43e42a0 2023-07-14T19:24:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa6f5ffb700 10 client.15222 handle_command_reply: tid=0 2023-07-14T19:24:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 client.15222 shutdown 2023-07-14T19:24:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 -- 172.21.15.53:0/172541842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 msgr2=0x560dc43835d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc43835d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa704011770 tx=0x7fa7040094a0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 -- 172.21.15.53:0/172541842 shutdown_connections 2023-07-14T19:24:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.70:6836/3730963239,v1:172.21.15.70:6837/3730963239] conn(0x560dc43e42a0 0x560dc43e0c30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560dc4377460 0x560dc4385ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa6f5ffb700 0 client.15222 ms_handle_reset on v2:172.21.15.70:6836/3730963239 2023-07-14T19:24:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x560dc4375f10 0x560dc4383090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 --2- 172.21.15.53:0/172541842 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560dc4376860 0x560dc43835d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 -- 172.21.15.53:0/172541842 >> 172.21.15.53:0/172541842 conn(0x560dc436b630 msgr2=0x560dc4389eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.047+0000 7fa720ce0b80 1 -- 172.21.15.53:0/172541842 shutdown_connections 2023-07-14T19:24:55.060 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa720ce0b80 1 -- 172.21.15.53:0/172541842 wait complete. 2023-07-14T19:24:55.060 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa720ce0b80 20 client.15222 trim_cache size 0 max 16384 2023-07-14T19:24:55.060 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-07-14T19:24:55.060 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:24:55.061 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 1, 2023-07-14T19:24:55.061 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24491, 2023-07-14T19:24:55.061 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2023-07-14T19:24:55.061 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2023-07-14T19:24:55.061 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2023-07-14T19:24:55.061 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 73.581635315, 2023-07-14T19:24:55.061 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 23, 2023-07-14T19:24:55.061 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 82, 2023-07-14T19:24:55.062 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 80, 2023-07-14T19:24:55.062 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 92.221405411999996 2023-07-14T19:24:55.062 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-07-14T19:24:55.063 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 -- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa6f80a11e0 msgr2=0x7fa6f80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.063 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa6f80a11e0 0x7fa6f80a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa704009bf0 tx=0x7fa704009260 comp rx=0 tx=0).stop 2023-07-14T19:24:55.063 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 -- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 msgr2=0x7fa7101947f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.063 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 0x7fa7101947f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa7000098e0 tx=0x7fa700005e80 comp rx=0 tx=0).stop 2023-07-14T19:24:55.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 -- 172.21.15.53:0/3542797872 shutdown_connections 2023-07-14T19:24:55.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fa6f80a11e0 0x7fa6f80a3690 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa710100690 0x7fa71019bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7100feb10 0x7fa7101947f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 --2- 172.21.15.53:0/3542797872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7100ffd00 0x7fa710194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 -- 172.21.15.53:0/3542797872 >> 172.21.15.53:0/3542797872 conn(0x7fa7100fa130 msgr2=0x7fa7100fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.064 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 -- 172.21.15.53:0/3542797872 shutdown_connections 2023-07-14T19:24:55.065 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.048+0000 7fa714b48700 1 -- 172.21.15.53:0/3542797872 wait complete. 2023-07-14T19:24:55.065 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T19:24:55.191 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.190+0000 7f6e28f52700 1 -- 172.21.15.53:0/2543541049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e24100690 msgr2=0x7f6e24106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.192 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.190+0000 7f6e28f52700 1 --2- 172.21.15.53:0/2543541049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e24100690 0x7f6e24106f50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00b0a0 tx=0x7f6e1c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.192 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.190+0000 7f6e28f52700 1 -- 172.21.15.53:0/2543541049 shutdown_connections 2023-07-14T19:24:55.192 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.190+0000 7f6e28f52700 1 --2- 172.21.15.53:0/2543541049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e24100690 0x7f6e24106f50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.192 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.190+0000 7f6e28f52700 1 --2- 172.21.15.53:0/2543541049 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e240feb10 0x7f6e240fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.192 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.190+0000 7f6e28f52700 1 --2- 172.21.15.53:0/2543541049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e240ffd00 0x7f6e24100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.193 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.190+0000 7f6e28f52700 1 -- 172.21.15.53:0/2543541049 >> 172.21.15.53:0/2543541049 conn(0x7f6e240fa130 msgr2=0x7f6e240fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.193 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.190+0000 7f6e28f52700 1 -- 172.21.15.53:0/2543541049 shutdown_connections 2023-07-14T19:24:55.193 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.190+0000 7f6e28f52700 1 -- 172.21.15.53:0/2543541049 wait complete. 2023-07-14T19:24:55.194 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e28f52700 1 Processor -- start 2023-07-14T19:24:55.195 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e28f52700 1 -- start start 2023-07-14T19:24:55.195 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e28f52700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e240feb10 0x7f6e24102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.195 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e28f52700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e240ffd00 0x7f6e241010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.195 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e28f52700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e24100690 0x7f6e24101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.195 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e28f52700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24101dc0 con 0x7f6e240feb10 2023-07-14T19:24:55.195 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e28f52700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e24101f30 con 0x7f6e240ffd00 2023-07-14T19:24:55.195 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e28f52700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2419fb80 con 0x7f6e24100690 2023-07-14T19:24:55.196 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e2259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e240feb10 0x7f6e24102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.196 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e2259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e240feb10 0x7f6e24102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51464/0 (socket says 172.21.15.53:51464) 2023-07-14T19:24:55.196 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.191+0000 7f6e2259c700 1 -- 172.21.15.53:0/3294839614 learned_addr learned my addr 172.21.15.53:0/3294839614 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.196 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e240ffd00 0x7f6e241010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.196 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e22d9d700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e24100690 0x7f6e24101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.196 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e2259c700 1 -- 172.21.15.53:0/3294839614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e24100690 msgr2=0x7f6e24101620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.196 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e2259c700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e24100690 0x7f6e24101620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.197 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e2259c700 1 -- 172.21.15.53:0/3294839614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e240ffd00 msgr2=0x7f6e241010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.197 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e2259c700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e240ffd00 0x7f6e241010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.197 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e2259c700 1 -- 172.21.15.53:0/3294839614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c009d00 con 0x7f6e240feb10 2023-07-14T19:24:55.197 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e240ffd00 0x7f6e241010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:55.197 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e2259c700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e240feb10 0x7f6e24102a50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400d980 tx=0x7f6e1400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.197 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e137fe700 1 -- 172.21.15.53:0/3294839614 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14009ae0 con 0x7f6e240feb10 2023-07-14T19:24:55.197 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e137fe700 1 -- 172.21.15.53:0/3294839614 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e1400de60 con 0x7f6e240feb10 2023-07-14T19:24:55.197 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e28f52700 1 -- 172.21.15.53:0/3294839614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2419fe10 con 0x7f6e240feb10 2023-07-14T19:24:55.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e137fe700 1 -- 172.21.15.53:0/3294839614 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1400f620 con 0x7f6e240feb10 2023-07-14T19:24:55.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.192+0000 7f6e28f52700 1 -- 172.21.15.53:0/3294839614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e241a01e0 con 0x7f6e240feb10 2023-07-14T19:24:55.201 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.195+0000 7f6e137fe700 1 -- 172.21.15.53:0/3294839614 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6e14010460 con 0x7f6e240feb10 2023-07-14T19:24:55.202 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e137fe700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6e0c0a11e0 0x7f6e0c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.202 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e350eab80 1 Processor -- start 2023-07-14T19:24:55.202 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6e0c0a11e0 0x7f6e0c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.202 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e137fe700 1 -- 172.21.15.53:0/3294839614 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f6e140cc6d0 con 0x7f6e240feb10 2023-07-14T19:24:55.202 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e350eab80 1 -- start start 2023-07-14T19:24:55.202 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.202 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b7860 0x55bed44b7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e350eab80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed43bbc60 con 0x55bed44b6f10 2023-07-14T19:24:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e350eab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed440e9f0 con 0x55bed44b8460 2023-07-14T19:24:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e350eab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed4407e80 con 0x55bed44b7860 2023-07-14T19:24:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e2259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6e0c0a11e0 0x7f6e0c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6e18005950 tx=0x7f6e180058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e2259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44b7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51472/0 (socket says 172.21.15.53:51472) 2023-07-14T19:24:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.196+0000 7f6e2259c700 1 -- 172.21.15.53:0/501183831 learned_addr learned my addr 172.21.15.53:0/501183831 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b7860 0x55bed44b7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e22d9d700 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e2259c700 1 -- 172.21.15.53:0/501183831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b7860 msgr2=0x55bed44b7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e2259c700 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b7860 0x55bed44b7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e2259c700 1 -- 172.21.15.53:0/501183831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 msgr2=0x55bed44b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e2259c700 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44b88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e2259c700 1 -- 172.21.15.53:0/501183831 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bed4235d90 con 0x55bed44b6f10 2023-07-14T19:24:55.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e22d9d700 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:55.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e2259c700 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44b7320 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6e140963c0 tx=0x7f6e140995c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8759f13fa675913c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e10ff9700 1 -- 172.21.15.53:0/501183831 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e140a1da0 con 0x55bed44b6f10 2023-07-14T19:24:55.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e10ff9700 1 -- 172.21.15.53:0/501183831 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e14097cf0 con 0x55bed44b6f10 2023-07-14T19:24:55.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.197+0000 7f6e10ff9700 1 -- 172.21.15.53:0/501183831 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e140990a0 con 0x55bed44b6f10 2023-07-14T19:24:55.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 -- 172.21.15.53:0/501183831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 msgr2=0x55bed44b7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44b7320 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6e140963c0 tx=0x7f6e140995c0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.205 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 -- 172.21.15.53:0/501183831 shutdown_connections 2023-07-14T19:24:55.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44b7320 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 --2- 172.21.15.53:0/501183831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b7860 0x55bed44b7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 -- 172.21.15.53:0/501183831 >> 172.21.15.53:0/501183831 conn(0x55bed44ac630 msgr2=0x55bed44b1980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 -- 172.21.15.53:0/501183831 shutdown_connections 2023-07-14T19:24:55.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 -- 172.21.15.53:0/501183831 wait complete. 2023-07-14T19:24:55.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 -- start start 2023-07-14T19:24:55.206 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.198+0000 7f6e350eab80 1 Processor -- start 2023-07-14T19:24:55.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44c0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b7860 0x55bed44c0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 0x55bed44c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed43a7a70 con 0x55bed44b6f10 2023-07-14T19:24:55.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed43a7770 con 0x55bed44b7860 2023-07-14T19:24:55.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed428b8d0 con 0x55bed44b8460 2023-07-14T19:24:55.207 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e2259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44c0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 10 client.0 _ll_register_callbacks cb 0x55bed43a5710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:55.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e2259c700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44c0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51478/0 (socket says 172.21.15.53:51478) 2023-07-14T19:24:55.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e2259c700 1 -- 172.21.15.53:0/1268057828 learned_addr learned my addr 172.21.15.53:0/1268057828 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e22d9d700 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 0x55bed44c7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b7860 0x55bed44c0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e2259c700 1 -- 172.21.15.53:0/1268057828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 msgr2=0x55bed44c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e2259c700 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 0x55bed44c7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.208 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e2259c700 1 -- 172.21.15.53:0/1268057828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b7860 msgr2=0x55bed44c0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e2259c700 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b7860 0x55bed44c0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e2259c700 1 -- 172.21.15.53:0/1268057828 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e241a01e0 con 0x55bed44b6f10 2023-07-14T19:24:55.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e22d9d700 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 0x55bed44c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:55.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e2259c700 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44c0990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6e140998e0 tx=0x7f6e1400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38a5b091296a08b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e09ffb700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:55.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1268057828 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14099e00 con 0x55bed44b6f10 2023-07-14T19:24:55.209 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1268057828 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f6e140a1da0 con 0x55bed44b6f10 2023-07-14T19:24:55.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 1 -- 172.21.15.53:0/1268057828 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bed43614a0 con 0x55bed44b6f10 2023-07-14T19:24:55.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1268057828 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e14009c40 con 0x55bed44b6f10 2023-07-14T19:24:55.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 10 client.15234 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:55.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.199+0000 7f6e350eab80 1 -- 172.21.15.53:0/1268057828 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bed421ec10 con 0x55bed44b6f10 2023-07-14T19:24:55.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1268057828 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e14063020 con 0x55bed44b6f10 2023-07-14T19:24:55.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1268057828 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f6e140958d0 con 0x55bed44b6f10 2023-07-14T19:24:55.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e350eab80 10 client.15234 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:55.210 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e350eab80 10 client.15234 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:24:55.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e350eab80 20 client.15234 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:55.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x55bed44cb720 0x55bed44cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e350eab80 4 client.15234 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:24:55.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e350eab80 1 -- 172.21.15.53:0/1268057828 --> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bed4485d50 con 0x55bed44cb720 2023-07-14T19:24:55.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x55bed44cb720 0x55bed44cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x55bed44cb720 0x55bed44cadb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76932da896ea2e14 server_cookie=3b819fec084ebe8e in_seq=0 out_seq=0 2023-07-14T19:24:55.211 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.200+0000 7f6e09ffb700 10 client.15234 ms_handle_connect on v2:172.21.15.70:6838/2496341573 2023-07-14T19:24:55.212 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1268057828 <== mds.0 v2:172.21.15.70:6838/2496341573 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55bed4485d50 con 0x55bed44cb720 2023-07-14T19:24:55.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e09ffb700 10 client.15234 handle_command_reply: tid=0 2023-07-14T19:24:55.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 client.15234 shutdown 2023-07-14T19:24:55.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 -- 172.21.15.53:0/1268057828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 msgr2=0x55bed44c0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44c0990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6e140998e0 tx=0x7f6e1400bf90 comp rx=0 tx=0).stop 2023-07-14T19:24:55.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 -- 172.21.15.53:0/1268057828 shutdown_connections 2023-07-14T19:24:55.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x55bed44cb720 0x55bed44cadb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 0x55bed44c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e09ffb700 0 client.15234 ms_handle_reset on v2:172.21.15.70:6838/2496341573 2023-07-14T19:24:55.214 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b6f10 0x55bed44c0990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.214 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1268057828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b7860 0x55bed44c0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.214 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 -- 172.21.15.53:0/1268057828 >> 172.21.15.53:0/1268057828 conn(0x55bed44ac630 msgr2=0x55bed44b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.214 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 -- 172.21.15.53:0/1268057828 shutdown_connections 2023-07-14T19:24:55.214 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 1 -- 172.21.15.53:0/1268057828 wait complete. 2023-07-14T19:24:55.214 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.201+0000 7f6e350eab80 20 client.15234 trim_cache size 0 max 16384 2023-07-14T19:24:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e350eab80 1 Processor -- start 2023-07-14T19:24:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e350eab80 1 -- start start 2023-07-14T19:24:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b6f10 0x55bed44b8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e350eab80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed44bfe90 con 0x55bed44b7860 2023-07-14T19:24:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e350eab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed44b2e60 con 0x55bed44b8460 2023-07-14T19:24:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e350eab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed44ae6d0 con 0x55bed44b6f10 2023-07-14T19:24:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e21d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e21d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51488/0 (socket says 172.21.15.53:51488) 2023-07-14T19:24:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e21d9b700 1 -- 172.21.15.53:0/3003549606 learned_addr learned my addr 172.21.15.53:0/3003549606 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e22d9d700 1 --2- 172.21.15.53:0/3003549606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e21d9b700 1 -- 172.21.15.53:0/3003549606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b6f10 msgr2=0x55bed44b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/3003549606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b6f10 0x55bed44b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e21d9b700 1 -- 172.21.15.53:0/3003549606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 msgr2=0x55bed44cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/3003549606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.219+0000 7f6e21d9b700 1 -- 172.21.15.53:0/3003549606 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bed425c200 con 0x55bed44b7860 2023-07-14T19:24:55.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e22d9d700 1 --2- 172.21.15.53:0/3003549606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:55.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/3003549606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c7ed0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800b5d0 tx=0x7f6e18012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d7d724d77d8066 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e117fa700 1 -- 172.21.15.53:0/3003549606 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e18017040 con 0x55bed44b7860 2023-07-14T19:24:55.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e117fa700 1 -- 172.21.15.53:0/3003549606 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e18003d40 con 0x55bed44b7860 2023-07-14T19:24:55.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e117fa700 1 -- 172.21.15.53:0/3003549606 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e18011e30 con 0x55bed44b7860 2023-07-14T19:24:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e350eab80 1 -- 172.21.15.53:0/3003549606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 msgr2=0x55bed44c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e350eab80 1 --2- 172.21.15.53:0/3003549606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c7ed0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800b5d0 tx=0x7f6e18012000 comp rx=0 tx=0).stop 2023-07-14T19:24:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e350eab80 1 -- 172.21.15.53:0/3003549606 shutdown_connections 2023-07-14T19:24:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e350eab80 1 --2- 172.21.15.53:0/3003549606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b8460 0x55bed44cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e350eab80 1 --2- 172.21.15.53:0/3003549606 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b6f10 0x55bed44b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e350eab80 1 --2- 172.21.15.53:0/3003549606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c7ed0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e350eab80 1 -- 172.21.15.53:0/3003549606 >> 172.21.15.53:0/3003549606 conn(0x55bed44ac630 msgr2=0x55bed44adb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e350eab80 1 -- 172.21.15.53:0/3003549606 shutdown_connections 2023-07-14T19:24:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.220+0000 7f6e350eab80 1 -- 172.21.15.53:0/3003549606 wait complete. 2023-07-14T19:24:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e350eab80 1 -- start start 2023-07-14T19:24:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e350eab80 1 Processor -- start 2023-07-14T19:24:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b6f10 0x55bed44c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e350eab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 0x55bed451f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e350eab80 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed451f8a0 con 0x55bed44b7860 2023-07-14T19:24:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e350eab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed451f9e0 con 0x55bed44b6f10 2023-07-14T19:24:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e350eab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bed451fb20 con 0x55bed44b8460 2023-07-14T19:24:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e350eab80 10 client.0 _ll_register_callbacks cb 0x7f6e140992b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:24:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e21d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e21d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51494/0 (socket says 172.21.15.53:51494) 2023-07-14T19:24:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e21d9b700 1 -- 172.21.15.53:0/1509962838 learned_addr learned my addr 172.21.15.53:0/1509962838 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e22d9d700 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 0x55bed451f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e21d9b700 1 -- 172.21.15.53:0/1509962838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 msgr2=0x55bed451f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 0x55bed451f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e21d9b700 1 -- 172.21.15.53:0/1509962838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b6f10 msgr2=0x55bed44c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:24:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e2259c700 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b6f10 0x55bed44c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b6f10 0x55bed44c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e21d9b700 1 -- 172.21.15.53:0/1509962838 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1800bdd0 con 0x55bed44b7860 2023-07-14T19:24:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.221+0000 7f6e2259c700 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b6f10 0x55bed44c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e21d9b700 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c5690 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800bda0 tx=0x7f6e18011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffd37c732bf53c3 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e09ffb700 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2023-07-14T19:24:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1509962838 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e18003680 con 0x55bed44b7860 2023-07-14T19:24:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1509962838 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f6e18017070 con 0x55bed44b7860 2023-07-14T19:24:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e350eab80 1 -- 172.21.15.53:0/1509962838 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e14097cf0 con 0x55bed44b7860 2023-07-14T19:24:55.232 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1509962838 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e180037e0 con 0x55bed44b7860 2023-07-14T19:24:55.232 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e350eab80 10 client.15240 fetch_fsmap learned FSMap version 23 2023-07-14T19:24:55.232 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e350eab80 1 -- 172.21.15.53:0/1509962838 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bed4234260 con 0x55bed44b7860 2023-07-14T19:24:55.232 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1509962838 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e14097cf0 con 0x55bed44b7860 2023-07-14T19:24:55.232 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1509962838 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 23) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f6e18021420 con 0x55bed44b7860 2023-07-14T19:24:55.232 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e350eab80 10 client.15240 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:24:55.232 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e350eab80 10 client.15240 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:24:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e350eab80 20 client.15240 populate_metadata read hostname 'smithi053' 2023-07-14T19:24:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x55bed4524750 0x55bed4521120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e350eab80 4 client.15240 mds_command: new command op to 24527 tid=0[{"prefix": "status"}] 2023-07-14T19:24:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.222+0000 7f6e350eab80 1 -- 172.21.15.53:0/1509962838 --> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bed4400e60 con 0x55bed4524750 2023-07-14T19:24:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.223+0000 7f6e2259c700 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x55bed4524750 0x55bed4521120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.223+0000 7f6e09ffb700 10 client.15240 ms_handle_connect on v2:172.21.15.70:6838/2496341573 2023-07-14T19:24:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.223+0000 7f6e2259c700 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x55bed4524750 0x55bed4521120 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47515af2bf0fb159 server_cookie=e7b5677dce68e296 in_seq=0 out_seq=0 2023-07-14T19:24:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.223+0000 7f6e09ffb700 1 -- 172.21.15.53:0/1509962838 <== mds.0 v2:172.21.15.70:6838/2496341573 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55bed4400e60 con 0x55bed4524750 2023-07-14T19:24:55.234 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.223+0000 7f6e09ffb700 10 client.15240 handle_command_reply: tid=0 2023-07-14T19:24:55.234 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.223+0000 7f6e350eab80 1 client.15240 shutdown 2023-07-14T19:24:55.234 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 -- 172.21.15.53:0/1509962838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 msgr2=0x55bed44c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.234 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c5690 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800bda0 tx=0x7f6e18011750 comp rx=0 tx=0).stop 2023-07-14T19:24:55.234 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 -- 172.21.15.53:0/1509962838 shutdown_connections 2023-07-14T19:24:55.234 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.70:6838/2496341573,v1:172.21.15.70:6839/2496341573] conn(0x55bed4524750 0x55bed4521120 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.234 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55bed44b8460 0x55bed451f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.234 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bed44b6f10 0x55bed44c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.235 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 --2- 172.21.15.53:0/1509962838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55bed44b7860 0x55bed44c5690 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.235 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e09ffb700 0 client.15240 ms_handle_reset on v2:172.21.15.70:6838/2496341573 2023-07-14T19:24:55.235 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 -- 172.21.15.53:0/1509962838 >> 172.21.15.53:0/1509962838 conn(0x55bed44ac630 msgr2=0x55bed44cadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.235 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 -- 172.21.15.53:0/1509962838 shutdown_connections 2023-07-14T19:24:55.235 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 1 -- 172.21.15.53:0/1509962838 wait complete. 2023-07-14T19:24:55.235 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e350eab80 20 client.15240 trim_cache size 0 max 16384 2023-07-14T19:24:55.236 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2023-07-14T19:24:55.236 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c5147628-227a-11ee-9b33-001a4aab830c", 2023-07-14T19:24:55.236 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2023-07-14T19:24:55.236 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 24527, 2023-07-14T19:24:55.236 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2023-07-14T19:24:55.236 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2023-07-14T19:24:55.237 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2023-07-14T19:24:55.237 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 76.777515218000005, 2023-07-14T19:24:55.237 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 23, 2023-07-14T19:24:55.237 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 82, 2023-07-14T19:24:55.237 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 80, 2023-07-14T19:24:55.237 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 83.446023956000005 2023-07-14T19:24:55.237 INFO:teuthology.orchestra.run.smithi053.stdout:} 2023-07-14T19:24:55.238 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 -- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6e0c0a11e0 msgr2=0x7f6e0c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6e0c0a11e0 0x7f6e0c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6e18005950 tx=0x7f6e180058e0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 -- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e240feb10 msgr2=0x7f6e24102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e240feb10 0x7f6e24102a50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400d980 tx=0x7f6e1400dd40 comp rx=0 tx=0).stop 2023-07-14T19:24:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 -- 172.21.15.53:0/3294839614 shutdown_connections 2023-07-14T19:24:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6e0c0a11e0 0x7f6e0c0a3690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e24100690 0x7f6e24101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e240feb10 0x7f6e24102a50 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 --2- 172.21.15.53:0/3294839614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e240ffd00 0x7f6e241010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:55.240 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 -- 172.21.15.53:0/3294839614 >> 172.21.15.53:0/3294839614 conn(0x7f6e240fa130 msgr2=0x7f6e240677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:55.240 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 -- 172.21.15.53:0/3294839614 shutdown_connections 2023-07-14T19:24:55.240 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:55.224+0000 7f6e28f52700 1 -- 172.21.15.53:0/3294839614 wait complete. 2023-07-14T19:24:55.240 INFO:teuthology.run_tasks:Running task kclient... 2023-07-14T19:24:55.273 INFO:tasks.kclient:Mounting kernel clients... 2023-07-14T19:24:55.274 INFO:tasks.kclient:config is {'client.0': {}} 2023-07-14T19:24:55.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:55.275 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2023-07-14T19:24:55.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:55.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2023-07-14T19:24:55.364 INFO:teuthology.orchestra.run.smithi053.stderr:Cannot find device "ceph-brx" 2023-07-14T19:24:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:24:55.366 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2023-07-14T19:24:55.366 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T19:24:55.367 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T19:24:55.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid c5147628-227a-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T19:24:55.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:55 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1415863495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:55.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:55 smithi070 ceph-mon[98015]: pgmap v289: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 57 B/s, 1 objects/s recovering 2023-07-14T19:24:55.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:55 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1415863495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:55.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:55 smithi104 ceph-mon[98435]: pgmap v289: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 57 B/s, 1 objects/s recovering 2023-07-14T19:24:55.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:55 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1415863495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:24:55.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:55 smithi053 ceph-mon[96565]: pgmap v289: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 57 B/s, 1 objects/s recovering 2023-07-14T19:24:55.964 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:57.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.030+0000 7f3f68fad700 1 -- 172.21.15.53:0/853436797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 msgr2=0x7f3f641037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:57.033 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.030+0000 7f3f68fad700 1 --2- 172.21.15.53:0/853436797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 0x7f3f641037b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0099c0 tx=0x7f3f4c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.030+0000 7f3f68fad700 1 -- 172.21.15.53:0/853436797 shutdown_connections 2023-07-14T19:24:57.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.030+0000 7f3f68fad700 1 --2- 172.21.15.53:0/853436797 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f64103ee0 0x7f3f6410ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.030+0000 7f3f68fad700 1 --2- 172.21.15.53:0/853436797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f640fee40 0x7f3f640ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.030+0000 7f3f68fad700 1 --2- 172.21.15.53:0/853436797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 0x7f3f641037b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.030+0000 7f3f68fad700 1 -- 172.21.15.53:0/853436797 >> 172.21.15.53:0/853436797 conn(0x7f3f64075ca0 msgr2=0x7f3f640760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:57.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.030+0000 7f3f68fad700 1 -- 172.21.15.53:0/853436797 shutdown_connections 2023-07-14T19:24:57.034 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.030+0000 7f3f68fad700 1 -- 172.21.15.53:0/853436797 wait complete. 2023-07-14T19:24:57.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f68fad700 1 Processor -- start 2023-07-14T19:24:57.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f68fad700 1 -- start start 2023-07-14T19:24:57.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f68fad700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f640fee40 0x7f3f64197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:57.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f68fad700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 0x7f3f64198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:57.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f68fad700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f64103ee0 0x7f3f6419c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:57.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f68fad700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6419cf00 con 0x7f3f640ff7e0 2023-07-14T19:24:57.035 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f68fad700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6419d070 con 0x7f3f640fee40 2023-07-14T19:24:57.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f68fad700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6419d230 con 0x7f3f64103ee0 2023-07-14T19:24:57.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f61d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 0x7f3f64198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:57.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f61d9b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 0x7f3f64198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51508/0 (socket says 172.21.15.53:51508) 2023-07-14T19:24:57.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f61d9b700 1 -- 172.21.15.53:0/889549887 learned_addr learned my addr 172.21.15.53:0/889549887 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:57.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.031+0000 7f3f6259c700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f640fee40 0x7f3f64197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:57.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f62d9d700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f64103ee0 0x7f3f6419c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:57.036 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f61d9b700 1 -- 172.21.15.53:0/889549887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f64103ee0 msgr2=0x7f3f6419c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:57.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f61d9b700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f64103ee0 0x7f3f6419c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f61d9b700 1 -- 172.21.15.53:0/889549887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f640fee40 msgr2=0x7f3f64197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:57.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f61d9b700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f640fee40 0x7f3f64197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f61d9b700 1 -- 172.21.15.53:0/889549887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c0095f0 con 0x7f3f640ff7e0 2023-07-14T19:24:57.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f62d9d700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f64103ee0 0x7f3f6419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:57.037 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f61d9b700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 0x7f3f64198480 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b580 tx=0x7f3f4c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:57.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f5b7fe700 1 -- 172.21.15.53:0/889549887 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c01d070 con 0x7f3f640ff7e0 2023-07-14T19:24:57.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6419d4b0 con 0x7f3f640ff7e0 2023-07-14T19:24:57.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f5b7fe700 1 -- 172.21.15.53:0/889549887 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f4c005510 con 0x7f3f640ff7e0 2023-07-14T19:24:57.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f5b7fe700 1 -- 172.21.15.53:0/889549887 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c004b30 con 0x7f3f640ff7e0 2023-07-14T19:24:57.038 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.032+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f641a4d70 con 0x7f3f640ff7e0 2023-07-14T19:24:57.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.033+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6404e340 con 0x7f3f640ff7e0 2023-07-14T19:24:57.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.034+0000 7f3f5b7fe700 1 -- 172.21.15.53:0/889549887 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3f4c00fb80 con 0x7f3f640ff7e0 2023-07-14T19:24:57.039 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.035+0000 7f3f5b7fe700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3f500a11e0 0x7f3f500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:57.040 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.035+0000 7f3f6259c700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3f500a11e0 0x7f3f500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:57.040 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.035+0000 7f3f5b7fe700 1 -- 172.21.15.53:0/889549887 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f3f4c0cdd00 con 0x7f3f640ff7e0 2023-07-14T19:24:57.040 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.035+0000 7f3f6259c700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3f500a11e0 0x7f3f500a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009c60 tx=0x7f3f54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:57.040 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.038+0000 7f3f5b7fe700 1 -- 172.21.15.53:0/889549887 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3f4c097350 con 0x7f3f640ff7e0 2023-07-14T19:24:57.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.283+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3f641a4f00 con 0x7f3f640ff7e0 2023-07-14T19:24:57.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.283+0000 7f3f5b7fe700 1 -- 172.21.15.53:0/889549887 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3f4c026060 con 0x7f3f640ff7e0 2023-07-14T19:24:57.285 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-07-14T19:24:57.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.285+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3f500a11e0 msgr2=0x7f3f500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:57.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.285+0000 7f3f68fad700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3f500a11e0 0x7f3f500a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009c60 tx=0x7f3f54009260 comp rx=0 tx=0).stop 2023-07-14T19:24:57.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.285+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 msgr2=0x7f3f64198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:57.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.285+0000 7f3f68fad700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 0x7f3f64198480 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b580 tx=0x7f3f4c00f690 comp rx=0 tx=0).stop 2023-07-14T19:24:57.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.285+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 shutdown_connections 2023-07-14T19:24:57.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.285+0000 7f3f68fad700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f3f500a11e0 0x7f3f500a3690 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.287 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.285+0000 7f3f68fad700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3f64103ee0 0x7f3f6419c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.285+0000 7f3f68fad700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f640fee40 0x7f3f64197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.286+0000 7f3f68fad700 1 --2- 172.21.15.53:0/889549887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f640ff7e0 0x7f3f64198480 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:57.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.286+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 >> 172.21.15.53:0/889549887 conn(0x7f3f64075ca0 msgr2=0x7f3f64068990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:57.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.286+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 shutdown_connections 2023-07-14T19:24:57.288 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:57.286+0000 7f3f68fad700 1 -- 172.21.15.53:0/889549887 wait complete. 2023-07-14T19:24:57.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:57 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/889549887' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:24:57.613 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-14T19:24:57.613 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T19:24:57.613 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2023-07-14T19:24:57.614 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-14T19:24:57.614 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T19:24:57.614 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-14T19:24:57.614 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T19:24:57.614 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T19:24:57.614 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-14T19:24:57.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:57.615 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2023-07-14T19:24:57.635 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T19:24:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T19:24:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T19:24:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:24:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2023-07-14T19:24:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:24:57.636 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:24:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2023-07-14T19:24:57.636 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:24:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2023-07-14T19:24:57.636 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T19:24:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:24:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-07-14T19:24:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 41281sec preferred_lft 41281sec 2023-07-14T19:24:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link 2023-07-14T19:24:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:24:57.637 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:24:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:24:57.638 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T19:24:57.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:57.639 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2023-07-14T19:24:57.639 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2023-07-14T19:24:57.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add name ceph-brx type bridge 2023-07-14T19:24:57.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr flush dev ceph-brx 2023-07-14T19:24:57.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set ceph-brx up 2023-07-14T19:24:57.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T19:24:57.641 DEBUG:teuthology.orchestra.run.smithi053:> ') 2023-07-14T19:24:57.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:57 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/889549887' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:24:57.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:57 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/889549887' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:24:57.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:57.806 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T19:24:57.835 INFO:teuthology.orchestra.run.smithi053.stdout:1 2023-07-14T19:24:57.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:57.837 DEBUG:teuthology.orchestra.run.smithi053:> route 2023-07-14T19:24:57.906 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2023-07-14T19:24:57.906 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T19:24:57.906 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-07-14T19:24:57.906 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-07-14T19:24:57.907 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T19:24:57.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:57.907 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2023-07-14T19:24:57.908 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2023-07-14T19:24:57.908 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-07-14T19:24:57.908 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-07-14T19:24:57.909 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-07-14T19:24:57.909 DEBUG:teuthology.orchestra.run.smithi053:> ') 2023-07-14T19:24:58.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:58.178 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2023-07-14T19:24:58.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:58.196 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list-id 2023-07-14T19:24:58.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:58.254 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2023-07-14T19:24:58.254 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2023-07-14T19:24:58.255 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T19:24:58.255 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-07-14T19:24:58.255 DEBUG:teuthology.orchestra.run.smithi053:> ') 2023-07-14T19:24:58.339 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:24:58 smithi053 ceph-mon[96565]: pgmap v290: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 57 B/s, 0 objects/s recovering 2023-07-14T19:24:58.357 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-07-14T19:24:58.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:58.358 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2023-07-14T19:24:58.358 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2023-07-14T19:24:58.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-07-14T19:24:58.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T19:24:58.359 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-07-14T19:24:58.359 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-07-14T19:24:58.359 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-07-14T19:24:58.359 DEBUG:teuthology.orchestra.run.smithi053:> ') 2023-07-14T19:24:58.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:58.583 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2023-07-14T19:24:58.584 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2023-07-14T19:24:58.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2023-07-14T19:24:58.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T19:24:58.584 DEBUG:teuthology.orchestra.run.smithi053:> ') 2023-07-14T19:24:58.686 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:24:58.687 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-14T19:24:58.704 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-14T19:24:58.704 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:24:58.705 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-14T19:24:58.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:24:58 smithi070 ceph-mon[98015]: pgmap v290: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 57 B/s, 0 objects/s recovering 2023-07-14T19:24:58.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:24:58 smithi104 ceph-mon[98435]: pgmap v290: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 57 B/s, 0 objects/s recovering 2023-07-14T19:24:58.761 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-07-14T19:24:58.761 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-07-14T19:24:58.762 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/_nogroup/sv_1/92cbb091-800c-4964-9131-34fa4e818ed4 2023-07-14T19:24:58.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:58.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/_nogroup/sv_1/92cbb091-800c-4964-9131-34fa4e818ed4 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-07-14T19:24:58.854 INFO:teuthology.orchestra.run.smithi053.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-07-14T19:24:58.854 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2023-07-14T19:24:58.980 INFO:teuthology.orchestra.run.smithi053.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-07-14T19:24:58.980 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2023-07-14T19:24:58.981 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: resolved to: "172.21.15.53:3300,172.21.15.70:3300,172.21.15.104:3300" 2023-07-14T19:24:58.981 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: trying mount with new device syntax: 0@c5147628-227a-11ee-9b33-001a4aab830c.cephfs=/volumes/_nogroup/sv_1/92cbb091-800c-4964-9131-34fa4e818ed4 2023-07-14T19:24:58.981 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.53:3300/172.21.15.70:3300/172.21.15.104:3300" will pass to kernel 2023-07-14T19:24:58.981 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: v2 mount support check returned 2 2023-07-14T19:24:58.981 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-14T19:24:58.981 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.53:3300,172.21.15.70:3300,172.21.15.104:3300:/volumes/_nogroup/sv_1/92cbb091-800c-4964-9131-34fa4e818ed4 2023-07-14T19:24:58.981 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mds_namespace=cephfs,fsid=c5147628-227a-11ee-9b33-001a4aab830c" will pass to kernel 2023-07-14T19:24:58.986 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-07-14T19:24:58.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:58.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-14T19:24:59.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:59.053 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-14T19:24:59.107 INFO:teuthology.orchestra.run.smithi053.stdout:client24989 2023-07-14T19:24:59.107 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:24:59.108 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:24:59.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:59.127 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-14T19:24:59.218 INFO:teuthology.orchestra.run.smithi053.stdout:client24989 2023-07-14T19:24:59.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:59.219 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-14T19:24:59.269 INFO:teuthology.orchestra.run.smithi053.stdout:client24989 2023-07-14T19:24:59.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:24:59.270 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/c5147628-227a-11ee-9b33-001a4aab830c.client24989/status') 2023-07-14T19:24:59.329 INFO:teuthology.orchestra.run.smithi053.stdout:instance: client.24989 (3)172.21.15.53:0/3283835541 2023-07-14T19:24:59.329 INFO:teuthology.orchestra.run.smithi053.stdout:blocklisted: false 2023-07-14T19:24:59.330 INFO:teuthology.orchestra.run.smithi053.stderr:0+1 records in 2023-07-14T19:24:59.330 INFO:teuthology.orchestra.run.smithi053.stderr:0+1 records out 2023-07-14T19:24:59.330 INFO:teuthology.orchestra.run.smithi053.stderr:71 bytes copied, 2.4975e-05 s, 2.8 MB/s 2023-07-14T19:24:59.331 INFO:teuthology.run_tasks:Running task exec... 2023-07-14T19:24:59.346 INFO:teuthology.task.exec:Executing custom commands... 2023-07-14T19:24:59.346 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi053.front.sepia.ceph.com 2023-07-14T19:24:59.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2023-07-14T19:24:59.477 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.476+0000 7f8ade578700 1 -- 172.21.15.53:0/3810253518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 msgr2=0x7f8ad8107cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:59.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.476+0000 7f8ade578700 1 --2- 172.21.15.53:0/3810253518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 0x7f8ad8107cc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009ec0 tx=0x7f8ad000b330 comp rx=0 tx=0).stop 2023-07-14T19:24:59.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.476+0000 7f8ade578700 1 -- 172.21.15.53:0/3810253518 shutdown_connections 2023-07-14T19:24:59.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.476+0000 7f8ade578700 1 --2- 172.21.15.53:0/3810253518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 0x7f8ad8107cc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:59.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.476+0000 7f8ade578700 1 --2- 172.21.15.53:0/3810253518 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ad8068770 0x7f8ad81027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:59.478 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.476+0000 7f8ade578700 1 --2- 172.21.15.53:0/3810253518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad8102d00 0x7f8ad8105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:59.479 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.476+0000 7f8ade578700 1 -- 172.21.15.53:0/3810253518 >> 172.21.15.53:0/3810253518 conn(0x7f8ad80fa130 msgr2=0x7f8ad80fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:59.479 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.476+0000 7f8ade578700 1 -- 172.21.15.53:0/3810253518 shutdown_connections 2023-07-14T19:24:59.479 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.476+0000 7f8ade578700 1 -- 172.21.15.53:0/3810253518 wait complete. 2023-07-14T19:24:59.481 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8ade578700 1 Processor -- start 2023-07-14T19:24:59.481 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8ade578700 1 -- start start 2023-07-14T19:24:59.481 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8ade578700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ad8068770 0x7f8ad8195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:59.481 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8ade578700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad8102d00 0x7f8ad8195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:59.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8ade578700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 0x7f8ad8196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:59.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8ade578700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81974b0 con 0x7f8ad8105810 2023-07-14T19:24:59.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8ade578700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad8197620 con 0x7f8ad8102d00 2023-07-14T19:24:59.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8ade578700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81977e0 con 0x7f8ad8068770 2023-07-14T19:24:59.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8adcb15700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 0x7f8ad8196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:59.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8adcb15700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 0x7f8ad8196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51538/0 (socket says 172.21.15.53:51538) 2023-07-14T19:24:59.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.477+0000 7f8adcb15700 1 -- 172.21.15.53:0/2363484410 learned_addr learned my addr 172.21.15.53:0/2363484410 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:24:59.482 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8ad7fff700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ad8068770 0x7f8ad8195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:59.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8ad77fe700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad8102d00 0x7f8ad8195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:59.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8adcb15700 1 -- 172.21.15.53:0/2363484410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ad8068770 msgr2=0x7f8ad8195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:59.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8adcb15700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ad8068770 0x7f8ad8195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:59.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8adcb15700 1 -- 172.21.15.53:0/2363484410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad8102d00 msgr2=0x7f8ad8195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:59.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8adcb15700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad8102d00 0x7f8ad8195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:59.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8adcb15700 1 -- 172.21.15.53:0/2363484410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad0005e70 con 0x7f8ad8105810 2023-07-14T19:24:59.483 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8adcb15700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 0x7f8ad8196e50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009be0 tx=0x7f8ad0009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:59.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8ad57fa700 1 -- 172.21.15.53:0/2363484410 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad000e040 con 0x7f8ad8105810 2023-07-14T19:24:59.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad8197a60 con 0x7f8ad8105810 2023-07-14T19:24:59.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8ad57fa700 1 -- 172.21.15.53:0/2363484410 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ad0007e40 con 0x7f8ad8105810 2023-07-14T19:24:59.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8ad57fa700 1 -- 172.21.15.53:0/2363484410 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad001bb20 con 0x7f8ad8105810 2023-07-14T19:24:59.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.478+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad819e280 con 0x7f8ad8105810 2023-07-14T19:24:59.484 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.480+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad804e420 con 0x7f8ad8105810 2023-07-14T19:24:59.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.480+0000 7f8ad57fa700 1 -- 172.21.15.53:0/2363484410 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8ad001bc80 con 0x7f8ad8105810 2023-07-14T19:24:59.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.481+0000 7f8ad57fa700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8ac00a11e0 0x7f8ac00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:59.485 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.481+0000 7f8ad57fa700 1 -- 172.21.15.53:0/2363484410 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f8ad00ce710 con 0x7f8ad8105810 2023-07-14T19:24:59.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.481+0000 7f8ad7fff700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8ac00a11e0 0x7f8ac00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:59.486 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.481+0000 7f8ad7fff700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8ac00a11e0 0x7f8ac00a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8005950 tx=0x7f8ac800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:59.487 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.486+0000 7f8ad57fa700 1 -- 172.21.15.53:0/2363484410 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8ad0097dc0 con 0x7f8ad8105810 2023-07-14T19:24:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:24:59.750+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7f8ad819e500 con 0x7f8ad8105810 2023-07-14T19:25:00.342 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.340+0000 7f8ad57fa700 1 -- 172.21.15.53:0/2363484410 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8ad009b3e0 con 0x7f8ad8105810 2023-07-14T19:25:00.345 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.343+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8ac00a11e0 msgr2=0x7f8ac00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:00.345 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.343+0000 7f8ade578700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8ac00a11e0 0x7f8ac00a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8005950 tx=0x7f8ac800b410 comp rx=0 tx=0).stop 2023-07-14T19:25:00.345 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 msgr2=0x7f8ad8196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:00.345 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 0x7f8ad8196e50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009be0 tx=0x7f8ad0009c10 comp rx=0 tx=0).stop 2023-07-14T19:25:00.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 shutdown_connections 2023-07-14T19:25:00.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f8ac00a11e0 0x7f8ac00a3690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:00.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ad8105810 0x7f8ad8196e50 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:00.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8ad8068770 0x7f8ad8195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:00.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 --2- 172.21.15.53:0/2363484410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad8102d00 0x7f8ad8195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:00.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 >> 172.21.15.53:0/2363484410 conn(0x7f8ad80fa130 msgr2=0x7f8ad8101820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:00.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 shutdown_connections 2023-07-14T19:25:00.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.344+0000 7f8ade578700 1 -- 172.21.15.53:0/2363484410 wait complete. 2023-07-14T19:25:00.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup distributed 1' 2023-07-14T19:25:00.492 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 -- 172.21.15.53:0/826317929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34a0100770 msgr2=0x7f34a0107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:00.492 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 --2- 172.21.15.53:0/826317929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34a0100770 0x7f34a0107010 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3498009ec0 tx=0x7f349800b330 comp rx=0 tx=0).stop 2023-07-14T19:25:00.493 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 -- 172.21.15.53:0/826317929 shutdown_connections 2023-07-14T19:25:00.493 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 --2- 172.21.15.53:0/826317929 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34a0100770 0x7f34a0107010 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:00.493 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 --2- 172.21.15.53:0/826317929 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34a00feb30 0x7f34a00fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:00.494 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 --2- 172.21.15.53:0/826317929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34a00ffdc0 0x7f34a0100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:00.494 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 -- 172.21.15.53:0/826317929 >> 172.21.15.53:0/826317929 conn(0x7f34a00fa130 msgr2=0x7f34a00fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:00.494 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 -- 172.21.15.53:0/826317929 shutdown_connections 2023-07-14T19:25:00.494 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 -- 172.21.15.53:0/826317929 wait complete. 2023-07-14T19:25:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 Processor -- start 2023-07-14T19:25:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.487+0000 7f34a73a9700 1 -- start start 2023-07-14T19:25:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a73a9700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34a00feb30 0x7f34a01011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a73a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34a00ffdc0 0x7f34a01016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a73a9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34a0100770 0x7f34a0101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a73a9700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a019f970 con 0x7f34a00feb30 2023-07-14T19:25:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a73a9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a019fab0 con 0x7f34a00ffdc0 2023-07-14T19:25:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a73a9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a019fbf0 con 0x7f34a0100770 2023-07-14T19:25:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5145700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34a00feb30 0x7f34a01011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:00.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5145700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34a00feb30 0x7f34a01011a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51546/0 (socket says 172.21.15.53:51546) 2023-07-14T19:25:00.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5145700 1 -- 172.21.15.53:0/3100849693 learned_addr learned my addr 172.21.15.53:0/3100849693 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:00.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5946700 1 --2- 172.21.15.53:0/3100849693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34a0100770 0x7f34a0101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a4944700 1 --2- 172.21.15.53:0/3100849693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34a00ffdc0 0x7f34a01016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5145700 1 -- 172.21.15.53:0/3100849693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34a0100770 msgr2=0x7f34a0101c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5145700 1 --2- 172.21.15.53:0/3100849693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34a0100770 0x7f34a0101c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5145700 1 -- 172.21.15.53:0/3100849693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34a00ffdc0 msgr2=0x7f34a01016e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5145700 1 --2- 172.21.15.53:0/3100849693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34a00ffdc0 0x7f34a01016e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5145700 1 -- 172.21.15.53:0/3100849693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3498005e70 con 0x7f34a00feb30 2023-07-14T19:25:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a5145700 1 --2- 172.21.15.53:0/3100849693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34a00feb30 0x7f34a01011a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f349000d950 tx=0x7f349000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:00.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34967fc700 1 -- 172.21.15.53:0/3100849693 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349000f890 con 0x7f34a00feb30 2023-07-14T19:25:00.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34967fc700 1 -- 172.21.15.53:0/3100849693 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3490010460 con 0x7f34a00feb30 2023-07-14T19:25:00.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.488+0000 7f34a73a9700 1 -- 172.21.15.53:0/3100849693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a019fe80 con 0x7f34a00feb30 2023-07-14T19:25:00.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.489+0000 7f34967fc700 1 -- 172.21.15.53:0/3100849693 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349000e610 con 0x7f34a00feb30 2023-07-14T19:25:00.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.489+0000 7f34a73a9700 1 -- 172.21.15.53:0/3100849693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a01a02b0 con 0x7f34a00feb30 2023-07-14T19:25:00.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.489+0000 7f34a73a9700 1 -- 172.21.15.53:0/3100849693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a004e420 con 0x7f34a00feb30 2023-07-14T19:25:00.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.491+0000 7f34967fc700 1 -- 172.21.15.53:0/3100849693 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f34900108d0 con 0x7f34a00feb30 2023-07-14T19:25:00.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.491+0000 7f34967fc700 1 --2- 172.21.15.53:0/3100849693 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f348c0a1260 0x7f348c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:00.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.491+0000 7f34a4944700 1 --2- 172.21.15.53:0/3100849693 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f348c0a1260 0x7f348c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:00.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.491+0000 7f34967fc700 1 -- 172.21.15.53:0/3100849693 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f34900cc7a0 con 0x7f34a00feb30 2023-07-14T19:25:00.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.492+0000 7f34a4944700 1 --2- 172.21.15.53:0/3100849693 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f348c0a1260 0x7f348c0a3710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f34a01024a0 tx=0x7f349c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:00.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.495+0000 7f34967fc700 1 -- 172.21.15.53:0/3100849693 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3490095dc0 con 0x7f34a00feb30 2023-07-14T19:25:00.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:00 smithi053 ceph-mon[96565]: pgmap v291: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:25:00.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:00 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2363484410' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:25:00.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:00 smithi070 ceph-mon[98015]: pgmap v291: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:25:00.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:00 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2363484410' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:25:00.728 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.727+0000 7f34a73a9700 1 -- 172.21.15.53:0/3100849693 --> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "distributed", "pin_setting": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f34a0068180 con 0x7f348c0a1260 2023-07-14T19:25:00.730 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:00.729+0000 7f34967fc700 1 -- 172.21.15.53:0/3100849693 <== mgr.14152 v2:172.21.15.53:6800/3014344459 1 ==== mgr_command_reply(tid 0: -1 Operation not permitted for group '_nogroup' as it is an internal group.) v1 ==== 80+0+0 (secure 0 0 0) 0x7f34a0068180 con 0x7f348c0a1260 2023-07-14T19:25:00.732 INFO:teuthology.orchestra.run.smithi053.stderr:Error EPERM: Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T19:25:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:25:00.736 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup distributed 1'" 2023-07-14T19:25:00.805 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9d92ce77e42b460cb34a1053e5b0e77e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup distributed 1'" 2023-07-14T19:25:00.808 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-07-14T19:25:00.821 INFO:tasks.kclient:Unmounting kernel clients... 2023-07-14T19:25:00.822 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:25:00.822 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:25:00.827 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:00 smithi104 ceph-mon[98435]: pgmap v291: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:25:00.828 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:00 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2363484410' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:25:00.845 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-07-14T19:25:00.846 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:25:00.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:25:01.072 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.068+0000 7fc4a353b700 1 -- 172.21.15.53:0/3736480313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c100680 msgr2=0x7fc49c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.073 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.068+0000 7fc4a353b700 1 --2- 172.21.15.53:0/3736480313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c100680 0x7fc49c106f40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc49400b0a0 tx=0x7fc49400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.073 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.068+0000 7fc4a353b700 1 -- 172.21.15.53:0/3736480313 shutdown_connections 2023-07-14T19:25:01.073 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 --2- 172.21.15.53:0/3736480313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c100680 0x7fc49c106f40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.073 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 --2- 172.21.15.53:0/3736480313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0feb00 0x7fc49c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 --2- 172.21.15.53:0/3736480313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc49c0ffcf0 0x7fc49c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 -- 172.21.15.53:0/3736480313 >> 172.21.15.53:0/3736480313 conn(0x7fc49c0fa100 msgr2=0x7fc49c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:01.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 -- 172.21.15.53:0/3736480313 shutdown_connections 2023-07-14T19:25:01.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 -- 172.21.15.53:0/3736480313 wait complete. 2023-07-14T19:25:01.074 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 Processor -- start 2023-07-14T19:25:01.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 -- start start 2023-07-14T19:25:01.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0feb00 0x7fc49c102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c0ffcf0 0x7fc49c1010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc49c100680 0x7fc49c101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.075 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c101db0 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c101f20 con 0x7fc49c100680 2023-07-14T19:25:01.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.069+0000 7fc4a353b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49c19fa20 con 0x7fc49c0feb00 2023-07-14T19:25:01.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a0ad6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c0ffcf0 0x7fc49c1010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a0ad6700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c0ffcf0 0x7fc49c1010d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51562/0 (socket says 172.21.15.53:51562) 2023-07-14T19:25:01.076 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a0ad6700 1 -- 172.21.15.53:0/957098607 learned_addr learned my addr 172.21.15.53:0/957098607 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:01.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a12d7700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0feb00 0x7fc49c102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a1ad8700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc49c100680 0x7fc49c101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a0ad6700 1 -- 172.21.15.53:0/957098607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0feb00 msgr2=0x7fc49c102a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a0ad6700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0feb00 0x7fc49c102a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a0ad6700 1 -- 172.21.15.53:0/957098607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc49c100680 msgr2=0x7fc49c101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a0ad6700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc49c100680 0x7fc49c101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.078 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a0ad6700 1 -- 172.21.15.53:0/957098607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc494009d00 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.078 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:25:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:25:00.729+0000 7f4d96359700 -1 mgr.server reply reply (1) Operation not permitted Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T19:25:01.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a0ad6700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c0ffcf0 0x7fc49c1010d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00d950 tx=0x7fc48c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:01.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4927fc700 1 -- 172.21.15.53:0/957098607 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c00f890 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4927fc700 1 -- 172.21.15.53:0/957098607 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc48c010460 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49c19fd00 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4927fc700 1 -- 172.21.15.53:0/957098607 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c00e610 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.070+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49c1a01c0 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.071+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49c10a2e0 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.072+0000 7fc4927fc700 1 -- 172.21.15.53:0/957098607 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc48c0108d0 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.081 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.073+0000 7fc4927fc700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc4880a1220 0x7fc4880a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.073+0000 7fc4927fc700 1 -- 172.21.15.53:0/957098607 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fc48c0cc7c0 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.073+0000 7fc4a12d7700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc4880a1220 0x7fc4880a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.073+0000 7fc4a12d7700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc4880a1220 0x7fc4880a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0ffb50 tx=0x7fc498007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:01.082 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.076+0000 7fc4927fc700 1 -- 172.21.15.53:0/957098607 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc48c095de0 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.278 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.277+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc49c1a07d0 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.279+0000 7fc4927fc700 1 -- 172.21.15.53:0/957098607 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22459 (secure 0 0 0) 0x7fc48c016090 con 0x7fc49c0ffcf0 2023-07-14T19:25:01.280 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:25:01.281 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:24:42.310475+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:25:01.282 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc4880a1220 msgr2=0x7fc4880a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc4880a1220 0x7fc4880a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0ffb50 tx=0x7fc498007400 comp rx=0 tx=0).stop 2023-07-14T19:25:01.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c0ffcf0 msgr2=0x7fc49c1010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c0ffcf0 0x7fc49c1010d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00d950 tx=0x7fc48c00dc60 comp rx=0 tx=0).stop 2023-07-14T19:25:01.283 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 shutdown_connections 2023-07-14T19:25:01.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fc4880a1220 0x7fc4880a36d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc49c100680 0x7fc49c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc49c0feb00 0x7fc49c102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.284 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 --2- 172.21.15.53:0/957098607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc49c0ffcf0 0x7fc49c1010d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 >> 172.21.15.53:0/957098607 conn(0x7fc49c0fa100 msgr2=0x7fc49c067820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:01.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 shutdown_connections 2023-07-14T19:25:01.285 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.281+0000 7fc4a353b700 1 -- 172.21.15.53:0/957098607 wait complete. 2023-07-14T19:25:01.295 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:19:51.775369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:23:35.935538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:23:37.252372+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:25:01.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:25:01.432 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 -- 172.21.15.53:0/795978657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4010100690 msgr2=0x7f4010106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.433 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 --2- 172.21.15.53:0/795978657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4010100690 0x7f4010106f50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f400800b0a0 tx=0x7f400800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.433 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 -- 172.21.15.53:0/795978657 shutdown_connections 2023-07-14T19:25:01.433 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 --2- 172.21.15.53:0/795978657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4010100690 0x7f4010106f50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.434 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 --2- 172.21.15.53:0/795978657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40100feb10 0x7f40100fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.434 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 --2- 172.21.15.53:0/795978657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40100ffd00 0x7f4010100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.434 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 -- 172.21.15.53:0/795978657 >> 172.21.15.53:0/795978657 conn(0x7f40100fa130 msgr2=0x7f40100fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:01.434 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 -- 172.21.15.53:0/795978657 shutdown_connections 2023-07-14T19:25:01.434 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 -- 172.21.15.53:0/795978657 wait complete. 2023-07-14T19:25:01.435 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.431+0000 7f4017008700 1 Processor -- start 2023-07-14T19:25:01.435 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f4017008700 1 -- start start 2023-07-14T19:25:01.437 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f4017008700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40100feb10 0x7f4010068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.437 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f4017008700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40100ffd00 0x7f401019a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.437 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f4017008700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4010100690 0x7f401019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.438 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f4017008700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401019d600 con 0x7f40100ffd00 2023-07-14T19:25:01.438 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f4017008700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401019d770 con 0x7f40100feb10 2023-07-14T19:25:01.438 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f4017008700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401019d930 con 0x7f4010100690 2023-07-14T19:25:01.438 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40100ffd00 0x7f401019a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400ffff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40100ffd00 0x7f401019a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51576/0 (socket says 172.21.15.53:51576) 2023-07-14T19:25:01.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400ffff700 1 -- 172.21.15.53:0/534240266 learned_addr learned my addr 172.21.15.53:0/534240266 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:01.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f4014da4700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40100feb10 0x7f4010068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f40155a5700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4010100690 0x7f401019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.439 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400ffff700 1 -- 172.21.15.53:0/534240266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4010100690 msgr2=0x7f401019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.440 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400ffff700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4010100690 0x7f401019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.440 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400ffff700 1 -- 172.21.15.53:0/534240266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40100feb10 msgr2=0x7f4010068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.440 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400ffff700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40100feb10 0x7f4010068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.440 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400ffff700 1 -- 172.21.15.53:0/534240266 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4008009d00 con 0x7f40100ffd00 2023-07-14T19:25:01.440 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400ffff700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40100ffd00 0x7f401019a7e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00d950 tx=0x7f3ffc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:01.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400dffb700 1 -- 172.21.15.53:0/534240266 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc009b40 con 0x7f40100ffd00 2023-07-14T19:25:01.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.432+0000 7f400dffb700 1 -- 172.21.15.53:0/534240266 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ffc009ca0 con 0x7f40100ffd00 2023-07-14T19:25:01.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.433+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401019dc10 con 0x7f40100ffd00 2023-07-14T19:25:01.441 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.433+0000 7f400dffb700 1 -- 172.21.15.53:0/534240266 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc00f620 con 0x7f40100ffd00 2023-07-14T19:25:01.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.433+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401019e170 con 0x7f40100ffd00 2023-07-14T19:25:01.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.434+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4010106000 con 0x7f40100ffd00 2023-07-14T19:25:01.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.435+0000 7f400dffb700 1 -- 172.21.15.53:0/534240266 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3ffc010460 con 0x7f40100ffd00 2023-07-14T19:25:01.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.435+0000 7f400dffb700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f40000a1260 0x7f40000a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.442 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.435+0000 7f400dffb700 1 -- 172.21.15.53:0/534240266 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f3ffc0ccd50 con 0x7f40100ffd00 2023-07-14T19:25:01.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.435+0000 7f4014da4700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f40000a1260 0x7f40000a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.436+0000 7f4014da4700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f40000a1260 0x7f40000a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f40100ffb60 tx=0x7f4004009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:01.443 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.439+0000 7f400dffb700 1 -- 172.21.15.53:0/534240266 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3ffc096370 con 0x7f40100ffd00 2023-07-14T19:25:01.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:01 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2363484410' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:25:01.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:01 smithi053 ceph-mon[96565]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:25:01.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:01 smithi053 ceph-mon[96565]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "distributed", "pin_setting": "1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:01.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:01 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/957098607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:25:01.573 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:01 smithi053 ceph-mon[96565]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:25:01.641 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.640+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f401002cb30 con 0x7f40100ffd00 2023-07-14T19:25:01.642 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.640+0000 7f400dffb700 1 -- 172.21.15.53:0/534240266 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ffc0200a0 con 0x7f40100ffd00 2023-07-14T19:25:01.642 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2023-07-14T19:25:01.645 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.642+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f40000a1260 msgr2=0x7f40000a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.646 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.642+0000 7f4017008700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f40000a1260 0x7f40000a3710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f40100ffb60 tx=0x7f4004009260 comp rx=0 tx=0).stop 2023-07-14T19:25:01.646 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.642+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40100ffd00 msgr2=0x7f401019a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.646 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.642+0000 7f4017008700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40100ffd00 0x7f401019a7e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00d950 tx=0x7f3ffc00dd10 comp rx=0 tx=0).stop 2023-07-14T19:25:01.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.642+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 shutdown_connections 2023-07-14T19:25:01.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.642+0000 7f4017008700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f40000a1260 0x7f40000a3710 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.647 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.643+0000 7f4017008700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4010100690 0x7f401019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.643+0000 7f4017008700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40100feb10 0x7f4010068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.643+0000 7f4017008700 1 --2- 172.21.15.53:0/534240266 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40100ffd00 0x7f401019a7e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.643+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 >> 172.21.15.53:0/534240266 conn(0x7f40100fa130 msgr2=0x7f4010103140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:01.648 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.643+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 shutdown_connections 2023-07-14T19:25:01.649 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.643+0000 7f4017008700 1 -- 172.21.15.53:0/534240266 wait complete. 2023-07-14T19:25:01.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:25:01.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:01 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2363484410' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:25:01.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:01 smithi070 ceph-mon[98015]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:25:01.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:01 smithi070 ceph-mon[98015]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "distributed", "pin_setting": "1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:01.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:01 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/957098607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:25:01.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:01 smithi070 ceph-mon[98015]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:25:01.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:01 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2363484410' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:25:01.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:01 smithi104 ceph-mon[98435]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:25:01.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:01 smithi104 ceph-mon[98435]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "distributed", "pin_setting": "1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:01.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:01 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/957098607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:25:01.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:01 smithi104 ceph-mon[98435]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:25:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.784+0000 7fb336e6b700 1 -- 172.21.15.53:0/486535016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3301058d0 msgr2=0x7fb330107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.784+0000 7fb336e6b700 1 --2- 172.21.15.53:0/486535016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3301058d0 0x7fb330107d30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb32800b0a0 tx=0x7fb32800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.784+0000 7fb336e6b700 1 -- 172.21.15.53:0/486535016 shutdown_connections 2023-07-14T19:25:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.784+0000 7fb336e6b700 1 --2- 172.21.15.53:0/486535016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3301058d0 0x7fb330107d30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.784+0000 7fb336e6b700 1 --2- 172.21.15.53:0/486535016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb330100690 0x7fb330102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.784+0000 7fb336e6b700 1 --2- 172.21.15.53:0/486535016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb330102fb0 0x7fb330105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.784+0000 7fb336e6b700 1 -- 172.21.15.53:0/486535016 >> 172.21.15.53:0/486535016 conn(0x7fb3300fa130 msgr2=0x7fb3300fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.784+0000 7fb336e6b700 1 -- 172.21.15.53:0/486535016 shutdown_connections 2023-07-14T19:25:01.788 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.784+0000 7fb336e6b700 1 -- 172.21.15.53:0/486535016 wait complete. 2023-07-14T19:25:01.790 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb336e6b700 1 Processor -- start 2023-07-14T19:25:01.791 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb336e6b700 1 -- start start 2023-07-14T19:25:01.791 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb336e6b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb330100690 0x7fb330195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.791 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb336e6b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb330102fb0 0x7fb330195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.791 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb336e6b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3301058d0 0x7fb33019cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.791 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb336e6b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33019d5c0 con 0x7fb330100690 2023-07-14T19:25:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb336e6b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33019d700 con 0x7fb3301058d0 2023-07-14T19:25:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb336e6b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33019d890 con 0x7fb330102fb0 2023-07-14T19:25:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb334c07700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb330100690 0x7fb330195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb334c07700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb330100690 0x7fb330195a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51586/0 (socket says 172.21.15.53:51586) 2023-07-14T19:25:01.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb334c07700 1 -- 172.21.15.53:0/3749117840 learned_addr learned my addr 172.21.15.53:0/3749117840 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:01.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.785+0000 7fb335408700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3301058d0 0x7fb33019cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb32ffff700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb330102fb0 0x7fb330195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb334c07700 1 -- 172.21.15.53:0/3749117840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb330102fb0 msgr2=0x7fb330195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb334c07700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb330102fb0 0x7fb330195f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb334c07700 1 -- 172.21.15.53:0/3749117840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3301058d0 msgr2=0x7fb33019cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb334c07700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3301058d0 0x7fb33019cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb334c07700 1 -- 172.21.15.53:0/3749117840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb328009d00 con 0x7fb330100690 2023-07-14T19:25:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb334c07700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb330100690 0x7fb330195a40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb32400e9a0 tx=0x7fb32400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb32dffb700 1 -- 172.21.15.53:0/3749117840 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324009790 con 0x7fb330100690 2023-07-14T19:25:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb32dffb700 1 -- 172.21.15.53:0/3749117840 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb32400cbd0 con 0x7fb330100690 2023-07-14T19:25:01.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33019db70 con 0x7fb330100690 2023-07-14T19:25:01.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb32dffb700 1 -- 172.21.15.53:0/3749117840 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324018680 con 0x7fb330100690 2023-07-14T19:25:01.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.786+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33019e080 con 0x7fb330100690 2023-07-14T19:25:01.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.787+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3300fbca0 con 0x7fb330100690 2023-07-14T19:25:01.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.788+0000 7fb32dffb700 1 -- 172.21.15.53:0/3749117840 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb32400cd40 con 0x7fb330100690 2023-07-14T19:25:01.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.789+0000 7fb32dffb700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3200a11e0 0x7fb3200a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:01.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.789+0000 7fb32dffb700 1 -- 172.21.15.53:0/3749117840 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fb324014070 con 0x7fb330100690 2023-07-14T19:25:01.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.789+0000 7fb32ffff700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3200a11e0 0x7fb3200a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:01.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.789+0000 7fb32ffff700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3200a11e0 0x7fb3200a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009b30 tx=0x7fb31c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:01.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.792+0000 7fb32dffb700 1 -- 172.21.15.53:0/3749117840 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb3240965c0 con 0x7fb330100690 2023-07-14T19:25:01.995 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.994+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb33002cb90 con 0x7fb330100690 2023-07-14T19:25:01.996 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.994+0000 7fb32dffb700 1 -- 172.21.15.53:0/3749117840 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb324099be0 con 0x7fb330100690 2023-07-14T19:25:01.996 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2023-07-14T19:25:01.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3200a11e0 msgr2=0x7fb3200a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.998 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3200a11e0 0x7fb3200a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb31c009b30 tx=0x7fb31c009260 comp rx=0 tx=0).stop 2023-07-14T19:25:01.999 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb330100690 msgr2=0x7fb330195a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:01.999 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb330100690 0x7fb330195a40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb32400e9a0 tx=0x7fb32400ecb0 comp rx=0 tx=0).stop 2023-07-14T19:25:01.999 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 shutdown_connections 2023-07-14T19:25:01.999 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb3200a11e0 0x7fb3200a3690 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.000 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3301058d0 0x7fb33019cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.000 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb330100690 0x7fb330195a40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.000 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 --2- 172.21.15.53:0/3749117840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb330102fb0 0x7fb330195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.000 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 >> 172.21.15.53:0/3749117840 conn(0x7fb3300fa130 msgr2=0x7fb3301012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:02.000 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 shutdown_connections 2023-07-14T19:25:02.001 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:01.997+0000 7fb336e6b700 1 -- 172.21.15.53:0/3749117840 wait complete. 2023-07-14T19:25:02.010 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:25:02.139 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.137+0000 7f9d9ada8700 1 -- 172.21.15.53:0/3720491115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d941006d0 msgr2=0x7f9d94102ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.140 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.137+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/3720491115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d941006d0 0x7f9d94102ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9d88009990 tx=0x7f9d88009ca0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.140 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.137+0000 7f9d9ada8700 1 -- 172.21.15.53:0/3720491115 shutdown_connections 2023-07-14T19:25:02.140 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.137+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/3720491115 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d94105910 0x7f9d94107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.140 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.137+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/3720491115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d941006d0 0x7f9d94102ab0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.141 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.137+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/3720491115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d94102ff0 0x7f9d941053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.141 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.137+0000 7f9d9ada8700 1 -- 172.21.15.53:0/3720491115 >> 172.21.15.53:0/3720491115 conn(0x7f9d940fa130 msgr2=0x7f9d940fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:02.141 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 -- 172.21.15.53:0/3720491115 shutdown_connections 2023-07-14T19:25:02.141 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 -- 172.21.15.53:0/3720491115 wait complete. 2023-07-14T19:25:02.141 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 Processor -- start 2023-07-14T19:25:02.142 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 -- start start 2023-07-14T19:25:02.143 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d941006d0 0x7f9d94195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:02.144 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d94102ff0 0x7f9d94195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:02.144 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d94105910 0x7f9d9419cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:02.144 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9419d530 con 0x7f9d94102ff0 2023-07-14T19:25:02.144 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9419d670 con 0x7f9d941006d0 2023-07-14T19:25:02.145 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d9ada8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9419d800 con 0x7f9d94105910 2023-07-14T19:25:02.145 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.138+0000 7f9d93fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d94102ff0 0x7f9d94195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:02.145 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d93fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d94102ff0 0x7f9d94195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51600/0 (socket says 172.21.15.53:51600) 2023-07-14T19:25:02.145 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d93fff700 1 -- 172.21.15.53:0/2972323884 learned_addr learned my addr 172.21.15.53:0/2972323884 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:02.146 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d99345700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d94105910 0x7f9d9419cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:02.146 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d98b44700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d941006d0 0x7f9d94195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:02.146 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d93fff700 1 -- 172.21.15.53:0/2972323884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d94105910 msgr2=0x7f9d9419cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.146 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d93fff700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d94105910 0x7f9d9419cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.146 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d93fff700 1 -- 172.21.15.53:0/2972323884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d941006d0 msgr2=0x7f9d94195a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.147 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d93fff700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d941006d0 0x7f9d94195a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.148 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d93fff700 1 -- 172.21.15.53:0/2972323884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d880095f0 con 0x7f9d94102ff0 2023-07-14T19:25:02.148 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d93fff700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d94102ff0 0x7f9d94195f90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000e9a0 tx=0x7f9d8000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:02.148 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d91ffb700 1 -- 172.21.15.53:0/2972323884 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80009790 con 0x7f9d94102ff0 2023-07-14T19:25:02.148 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d91ffb700 1 -- 172.21.15.53:0/2972323884 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d8000cbd0 con 0x7f9d94102ff0 2023-07-14T19:25:02.149 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d9419dae0 con 0x7f9d94102ff0 2023-07-14T19:25:02.149 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d91ffb700 1 -- 172.21.15.53:0/2972323884 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d80018680 con 0x7f9d94102ff0 2023-07-14T19:25:02.149 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.139+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d9419dff0 con 0x7f9d94102ff0 2023-07-14T19:25:02.149 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.140+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d940fbca0 con 0x7f9d94102ff0 2023-07-14T19:25:02.149 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.141+0000 7f9d91ffb700 1 -- 172.21.15.53:0/2972323884 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9d8000cd40 con 0x7f9d94102ff0 2023-07-14T19:25:02.150 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.142+0000 7f9d91ffb700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9d840a11e0 0x7f9d840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:02.150 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.142+0000 7f9d91ffb700 1 -- 172.21.15.53:0/2972323884 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f9d80014070 con 0x7f9d94102ff0 2023-07-14T19:25:02.150 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.142+0000 7f9d98b44700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9d840a11e0 0x7f9d840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:02.150 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.142+0000 7f9d98b44700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9d840a11e0 0x7f9d840a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800b2e0 tx=0x7f9d88005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:02.151 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.145+0000 7f9d91ffb700 1 -- 172.21.15.53:0/2972323884 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9d800965c0 con 0x7f9d94102ff0 2023-07-14T19:25:02.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.344+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9d9402caf0 con 0x7f9d94102ff0 2023-07-14T19:25:02.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.345+0000 7f9d91ffb700 1 -- 172.21.15.53:0/2972323884 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9d80099be0 con 0x7f9d94102ff0 2023-07-14T19:25:02.347 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2023-07-14T19:25:02.349 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9d840a11e0 msgr2=0x7f9d840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.349 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9d840a11e0 0x7f9d840a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800b2e0 tx=0x7f9d88005fb0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.349 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d94102ff0 msgr2=0x7f9d94195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.350 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d94102ff0 0x7f9d94195f90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000e9a0 tx=0x7f9d8000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.350 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 shutdown_connections 2023-07-14T19:25:02.350 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f9d840a11e0 0x7f9d840a3690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.350 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d94105910 0x7f9d9419cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.350 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d941006d0 0x7f9d94195a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.351 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 --2- 172.21.15.53:0/2972323884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d94102ff0 0x7f9d94195f90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.351 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 >> 172.21.15.53:0/2972323884 conn(0x7f9d940fa130 msgr2=0x7f9d94101330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:02.351 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 shutdown_connections 2023-07-14T19:25:02.351 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.347+0000 7f9d9ada8700 1 -- 172.21.15.53:0/2972323884 wait complete. 2023-07-14T19:25:02.362 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T19:25:02.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.487+0000 7fb89a05b700 1 -- 172.21.15.53:0/1788323097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8940ffc40 msgr2=0x7fb894100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.487+0000 7fb89a05b700 1 --2- 172.21.15.53:0/1788323097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8940ffc40 0x7fb894100050 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb884009990 tx=0x7fb884009ca0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.489 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.487+0000 7fb89a05b700 1 -- 172.21.15.53:0/1788323097 shutdown_connections 2023-07-14T19:25:02.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.487+0000 7fb89a05b700 1 --2- 172.21.15.53:0/1788323097 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8941017c0 0x7fb894108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.487+0000 7fb89a05b700 1 --2- 172.21.15.53:0/1788323097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8940ffc40 0x7fb894100050 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.487+0000 7fb89a05b700 1 --2- 172.21.15.53:0/1788323097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb894100e30 0x7fb894101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.487+0000 7fb89a05b700 1 -- 172.21.15.53:0/1788323097 >> 172.21.15.53:0/1788323097 conn(0x7fb8940fb240 msgr2=0x7fb8940fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:02.490 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.487+0000 7fb89a05b700 1 -- 172.21.15.53:0/1788323097 shutdown_connections 2023-07-14T19:25:02.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.487+0000 7fb89a05b700 1 -- 172.21.15.53:0/1788323097 wait complete. 2023-07-14T19:25:02.491 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb89a05b700 1 Processor -- start 2023-07-14T19:25:02.494 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb89a05b700 1 -- start start 2023-07-14T19:25:02.494 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb89a05b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8940ffc40 0x7fb8940743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:02.495 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb89a05b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb894100e30 0x7fb894072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:02.495 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb89a05b700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8941017c0 0x7fb894072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:02.495 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb89a05b700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb894073710 con 0x7fb8941017c0 2023-07-14T19:25:02.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb89a05b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb894073880 con 0x7fb894100e30 2023-07-14T19:25:02.496 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb89a05b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89419f9e0 con 0x7fb8940ffc40 2023-07-14T19:25:02.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb893fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8941017c0 0x7fb894072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:02.497 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb893fff700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8941017c0 0x7fb894072f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51616/0 (socket says 172.21.15.53:51616) 2023-07-14T19:25:02.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb893fff700 1 -- 172.21.15.53:0/3849281348 learned_addr learned my addr 172.21.15.53:0/3849281348 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:02.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb892ffd700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb894100e30 0x7fb894072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:02.498 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.488+0000 7fb8937fe700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8940ffc40 0x7fb8940743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb893fff700 1 -- 172.21.15.53:0/3849281348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8940ffc40 msgr2=0x7fb8940743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb893fff700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8940ffc40 0x7fb8940743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb893fff700 1 -- 172.21.15.53:0/3849281348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb894100e30 msgr2=0x7fb894072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.500 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb893fff700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb894100e30 0x7fb894072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb893fff700 1 -- 172.21.15.53:0/3849281348 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c009520 con 0x7fb8941017c0 2023-07-14T19:25:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb8937fe700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8940ffc40 0x7fb8940743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:25:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb893fff700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8941017c0 0x7fb894072f70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00ef30 tx=0x7fb88c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:02.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb890ff9700 1 -- 172.21.15.53:0/3849281348 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c018070 con 0x7fb8941017c0 2023-07-14T19:25:02.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb890ff9700 1 -- 172.21.15.53:0/3849281348 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb88c00f040 con 0x7fb8941017c0 2023-07-14T19:25:02.502 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8840095f0 con 0x7fb8941017c0 2023-07-14T19:25:02.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb890ff9700 1 -- 172.21.15.53:0/3849281348 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c00cdb0 con 0x7fb8941017c0 2023-07-14T19:25:02.503 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.489+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89419ff90 con 0x7fb8941017c0 2023-07-14T19:25:02.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.490+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89410b410 con 0x7fb8941017c0 2023-07-14T19:25:02.504 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.491+0000 7fb890ff9700 1 -- 172.21.15.53:0/3849281348 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb88c003a00 con 0x7fb8941017c0 2023-07-14T19:25:02.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.492+0000 7fb890ff9700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb8800a11e0 0x7fb8800a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:02.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.492+0000 7fb8937fe700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb8800a11e0 0x7fb8800a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:02.505 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.492+0000 7fb890ff9700 1 -- 172.21.15.53:0/3849281348 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7fb88c0ccf70 con 0x7fb8941017c0 2023-07-14T19:25:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.492+0000 7fb8937fe700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb8800a11e0 0x7fb8800a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb88400b2e0 tx=0x7fb884005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:02.507 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.496+0000 7fb890ff9700 1 -- 172.21.15.53:0/3849281348 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb88c0966c0 con 0x7fb8941017c0 2023-07-14T19:25:02.713 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.712+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb89410b5a0 con 0x7fb8941017c0 2023-07-14T19:25:02.713 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.712+0000 7fb890ff9700 1 -- 172.21.15.53:0/3849281348 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v82) v1 ==== 81+0+585 (secure 0 0 0) 0x7fb88c099ce0 con 0x7fb8941017c0 2023-07-14T19:25:02.714 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/1700081881 2023-07-15T19:17:41.240106+0000 2023-07-14T19:25:02.714 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6801/2149831 2023-07-15T19:17:41.240106+0000 2023-07-14T19:25:02.714 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6800/2149831 2023-07-15T19:17:41.240106+0000 2023-07-14T19:25:02.715 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/4078421180 2023-07-15T19:17:41.240106+0000 2023-07-14T19:25:02.715 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6800/3434712469 2023-07-15T19:17:15.286956+0000 2023-07-14T19:25:02.715 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6801/3434712469 2023-07-15T19:17:15.286956+0000 2023-07-14T19:25:02.715 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/228763847 2023-07-15T19:17:15.286956+0000 2023-07-14T19:25:02.716 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/3718395334 2023-07-15T19:17:15.286956+0000 2023-07-14T19:25:02.716 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/3412815128 2023-07-15T19:17:15.286956+0000 2023-07-14T19:25:02.716 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:0/2206633343 2023-07-15T19:17:41.240106+0000 2023-07-14T19:25:02.717 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb8800a11e0 msgr2=0x7fb8800a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.717 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb8800a11e0 0x7fb8800a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb88400b2e0 tx=0x7fb884005fd0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8941017c0 msgr2=0x7fb894072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8941017c0 0x7fb894072f70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00ef30 tx=0x7fb88c00c3a0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 shutdown_connections 2023-07-14T19:25:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7fb8800a11e0 0x7fb8800a3690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8941017c0 0x7fb894072f70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8940ffc40 0x7fb8940743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 --2- 172.21.15.53:0/3849281348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb894100e30 0x7fb894072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 >> 172.21.15.53:0/3849281348 conn(0x7fb8940fb240 msgr2=0x7fb8940fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 shutdown_connections 2023-07-14T19:25:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:02.714+0000 7fb89a05b700 1 -- 172.21.15.53:0/3849281348 wait complete. 2023-07-14T19:25:02.720 INFO:teuthology.orchestra.run.smithi053.stderr:listed 10 entries 2023-07-14T19:25:02.726 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:02 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/534240266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:25:02.727 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:02 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3749117840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:25:02.727 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:02 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/2972323884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:25:02.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:02 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/534240266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:25:02.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:02 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3749117840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:25:02.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:02 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/2972323884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:25:02.729 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-07-14T19:25:02.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:25:02.730 DEBUG:teuthology.orchestra.run.smithi053:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-07-14T19:25:02.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:02 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/534240266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:25:02.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:02 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3749117840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:25:02.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:02 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/2972323884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:25:02.800 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2023-07-14T19:25:02.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:25:02.801 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-14T19:25:02.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:25:02.829 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2023-07-14T19:25:02.886 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-14T19:25:02.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:25:02.887 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T19:25:02.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:25:02.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2023-07-14T19:25:03.073 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-14T19:25:03.086 INFO:tasks.cephadm:Teardown begin 2023-07-14T19:25:03.086 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-07-14T19:25:03.087 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:25:03.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.211+0000 7f874a981700 1 -- 172.21.15.53:0/1042427938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87440ffcf0 msgr2=0x7f8744100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.214 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.211+0000 7f874a981700 1 --2- 172.21.15.53:0/1042427938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87440ffcf0 0x7f8744100140 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8738009990 tx=0x7f8738009ca0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.214 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.212+0000 7f874a981700 1 -- 172.21.15.53:0/1042427938 shutdown_connections 2023-07-14T19:25:03.214 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.212+0000 7f874a981700 1 --2- 172.21.15.53:0/1042427938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8744100680 0x7f8744106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.215 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.212+0000 7f874a981700 1 --2- 172.21.15.53:0/1042427938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87440feb00 0x7f87440fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.215 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.212+0000 7f874a981700 1 --2- 172.21.15.53:0/1042427938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87440ffcf0 0x7f8744100140 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.215 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.212+0000 7f874a981700 1 -- 172.21.15.53:0/1042427938 >> 172.21.15.53:0/1042427938 conn(0x7f87440fa100 msgr2=0x7f87440fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:03.215 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.212+0000 7f874a981700 1 -- 172.21.15.53:0/1042427938 shutdown_connections 2023-07-14T19:25:03.216 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.212+0000 7f874a981700 1 -- 172.21.15.53:0/1042427938 wait complete. 2023-07-14T19:25:03.216 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.212+0000 7f874a981700 1 Processor -- start 2023-07-14T19:25:03.216 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.212+0000 7f874a981700 1 -- start start 2023-07-14T19:25:03.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f874a981700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87440feb00 0x7f8744068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:03.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f874a981700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87440ffcf0 0x7f874419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:03.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f874a981700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8744100680 0x7f874419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:03.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f874a981700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874419d490 con 0x7f8744100680 2023-07-14T19:25:03.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f874a981700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874419d600 con 0x7f87440ffcf0 2023-07-14T19:25:03.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f874a981700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874419d7c0 con 0x7f87440feb00 2023-07-14T19:25:03.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8748f1e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8744100680 0x7f874419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:03.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f87437fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87440ffcf0 0x7f874419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:03.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8748f1e700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8744100680 0x7f874419ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51630/0 (socket says 172.21.15.53:51630) 2023-07-14T19:25:03.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f87437fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87440ffcf0 0x7f874419a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.53:38442/0 (socket says 172.21.15.53:38442) 2023-07-14T19:25:03.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8748f1e700 1 -- 172.21.15.53:0/814153170 learned_addr learned my addr 172.21.15.53:0/814153170 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:03.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8743fff700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87440feb00 0x7f8744068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:03.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8748f1e700 1 -- 172.21.15.53:0/814153170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87440feb00 msgr2=0x7f8744068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8748f1e700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87440feb00 0x7f8744068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8748f1e700 1 -- 172.21.15.53:0/814153170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87440ffcf0 msgr2=0x7f874419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8748f1e700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87440ffcf0 0x7f874419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8748f1e700 1 -- 172.21.15.53:0/814153170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380095f0 con 0x7f8744100680 2023-07-14T19:25:03.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f8748f1e700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8744100680 0x7f874419ccc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f873c00ef30 tx=0x7f873c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:03.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f87417fa700 1 -- 172.21.15.53:0/814153170 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c018070 con 0x7f8744100680 2023-07-14T19:25:03.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874419da40 con 0x7f8744100680 2023-07-14T19:25:03.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f87417fa700 1 -- 172.21.15.53:0/814153170 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f873c00f040 con 0x7f8744100680 2023-07-14T19:25:03.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f87417fa700 1 -- 172.21.15.53:0/814153170 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c00cd00 con 0x7f8744100680 2023-07-14T19:25:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.213+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874419df50 con 0x7f8744100680 2023-07-14T19:25:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.214+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8744106090 con 0x7f8744100680 2023-07-14T19:25:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.216+0000 7f87417fa700 1 -- 172.21.15.53:0/814153170 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f873c003a00 con 0x7f8744100680 2023-07-14T19:25:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.216+0000 7f87417fa700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f872c0a1220 0x7f872c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:03.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.216+0000 7f87417fa700 1 -- 172.21.15.53:0/814153170 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f873c0ccf40 con 0x7f8744100680 2023-07-14T19:25:03.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.216+0000 7f8743fff700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f872c0a1220 0x7f872c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:03.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.217+0000 7f8743fff700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f872c0a1220 0x7f872c0a36d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f87440ffb50 tx=0x7f8734009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:03.224 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.219+0000 7f87417fa700 1 -- 172.21.15.53:0/814153170 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f873c0965f0 con 0x7f8744100680 2023-07-14T19:25:03.467 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.466+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f874402c8e0 con 0x7f8744100680 2023-07-14T19:25:03.468 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.467+0000 7f87417fa700 1 -- 172.21.15.53:0/814153170 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f873c099c10 con 0x7f8744100680 2023-07-14T19:25:03.468 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:25:03.469 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:23:37.894565+0000","modified":"2023-07-14T19:25:00.337872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24491,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24484":{"gid":24484,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24491":{"gid":24491,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6839/2496341573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":2496341573},{"type":"v1","addr":"172.21.15.70:6839","nonce":2496341573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:25:03.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f872c0a1220 msgr2=0x7f872c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f872c0a1220 0x7f872c0a36d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f87440ffb50 tx=0x7f8734009260 comp rx=0 tx=0).stop 2023-07-14T19:25:03.470 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8744100680 msgr2=0x7f874419ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8744100680 0x7f874419ccc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f873c00ef30 tx=0x7f873c00c3a0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 shutdown_connections 2023-07-14T19:25:03.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f872c0a1220 0x7f872c0a36d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8744100680 0x7f874419ccc0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.471 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87440feb00 0x7f8744068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 --2- 172.21.15.53:0/814153170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87440ffcf0 0x7f874419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 >> 172.21.15.53:0/814153170 conn(0x7f87440fa100 msgr2=0x7f87441031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:03.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 shutdown_connections 2023-07-14T19:25:03.472 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.469+0000 7f874a981700 1 -- 172.21.15.53:0/814153170 wait complete. 2023-07-14T19:25:03.473 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 24 2023-07-14T19:25:03.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:25:03.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.607+0000 7f323dfae700 1 -- 172.21.15.53:0/427504619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 msgr2=0x7f32380fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.607+0000 7f323dfae700 1 --2- 172.21.15.53:0/427504619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 0x7f32380fe690 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3228009990 tx=0x7f3228009ca0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.609 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.607+0000 7f323dfae700 1 -- 172.21.15.53:0/427504619 shutdown_connections 2023-07-14T19:25:03.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.607+0000 7f323dfae700 1 --2- 172.21.15.53:0/427504619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32380ffe00 0x7f32381066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.607+0000 7f323dfae700 1 --2- 172.21.15.53:0/427504619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 0x7f32380fe690 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.607+0000 7f323dfae700 1 --2- 172.21.15.53:0/427504619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32380ff470 0x7f32380ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.607+0000 7f323dfae700 1 -- 172.21.15.53:0/427504619 >> 172.21.15.53:0/427504619 conn(0x7f32380f98a0 msgr2=0x7f32380fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:03.610 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.607+0000 7f323dfae700 1 -- 172.21.15.53:0/427504619 shutdown_connections 2023-07-14T19:25:03.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.607+0000 7f323dfae700 1 -- 172.21.15.53:0/427504619 wait complete. 2023-07-14T19:25:03.611 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f323dfae700 1 Processor -- start 2023-07-14T19:25:03.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f323dfae700 1 -- start start 2023-07-14T19:25:03.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f323dfae700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 0x7f3238195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:03.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f323dfae700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32380ff470 0x7f3238195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:03.613 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f323dfae700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32380ffe00 0x7f323819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f323dfae700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323819d3e0 con 0x7f32380fe280 2023-07-14T19:25:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f323dfae700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323819d520 con 0x7f32380ff470 2023-07-14T19:25:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f323dfae700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323819d6b0 con 0x7f32380ffe00 2023-07-14T19:25:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f32377fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 0x7f3238195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f32377fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 0x7f3238195860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51638/0 (socket says 172.21.15.53:51638) 2023-07-14T19:25:03.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f32377fe700 1 -- 172.21.15.53:0/1195054758 learned_addr learned my addr 172.21.15.53:0/1195054758 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:03.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.608+0000 7f3237fff700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32380ffe00 0x7f323819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:03.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f32377fe700 1 -- 172.21.15.53:0/1195054758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32380ffe00 msgr2=0x7f323819ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f32377fe700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32380ffe00 0x7f323819ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.615 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f32377fe700 1 -- 172.21.15.53:0/1195054758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32380ff470 msgr2=0x7f3238195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f32377fe700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32380ff470 0x7f3238195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f32377fe700 1 -- 172.21.15.53:0/1195054758 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32280095f0 con 0x7f32380fe280 2023-07-14T19:25:03.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f32377fe700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 0x7f3238195860 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3228000c00 tx=0x7f3228009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:03.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f3234ff9700 1 -- 172.21.15.53:0/1195054758 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322801d070 con 0x7f32380fe280 2023-07-14T19:25:03.616 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f3234ff9700 1 -- 172.21.15.53:0/1195054758 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3228004180 con 0x7f32380fe280 2023-07-14T19:25:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323819d930 con 0x7f32380fe280 2023-07-14T19:25:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f3234ff9700 1 -- 172.21.15.53:0/1195054758 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228022e00 con 0x7f32380fe280 2023-07-14T19:25:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.609+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323819de30 con 0x7f32380fe280 2023-07-14T19:25:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.610+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3238109220 con 0x7f32380fe280 2023-07-14T19:25:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.611+0000 7f3234ff9700 1 -- 172.21.15.53:0/1195054758 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3228022500 con 0x7f32380fe280 2023-07-14T19:25:03.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.611+0000 7f3234ff9700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f32240a12a0 0x7f32240a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:03.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.612+0000 7f3236ffd700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f32240a12a0 0x7f32240a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:03.618 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.612+0000 7f3234ff9700 1 -- 172.21.15.53:0/1195054758 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f32280ce0a0 con 0x7f32380fe280 2023-07-14T19:25:03.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.612+0000 7f3236ffd700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f32240a12a0 0x7f32240a3750 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3220005fd0 tx=0x7f3220005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:03.619 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.615+0000 7f3234ff9700 1 -- 172.21.15.53:0/1195054758 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f32280976f0 con 0x7f32380fe280 2023-07-14T19:25:03.727 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:03 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/3849281348' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:25:03.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:03 smithi070 ceph-mon[98015]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-07-14T19:25:03.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:03 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/3849281348' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:25:03.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:03 smithi104 ceph-mon[98435]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-07-14T19:25:03.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:03 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/3849281348' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:25:03.822 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:03 smithi053 ceph-mon[96565]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-07-14T19:25:03.862 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.861+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3238109410 con 0x7f32380fe280 2023-07-14T19:25:03.863 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.862+0000 7f3234ff9700 1 -- 172.21.15.53:0/1195054758 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10823 (secure 0 0 0) 0x7f322809ad10 con 0x7f32380fe280 2023-07-14T19:25:03.863 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:25:03.864 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2670019040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2670019040},{"type":"v1","addr":"172.21.15.53:6835","nonce":2670019040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1144651849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1144651849},{"type":"v1","addr":"172.21.15.53:6837","nonce":1144651849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6839/122790216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6838","nonce":122790216},{"type":"v1","addr":"172.21.15.53:6839","nonce":122790216}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6841/2523933697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6840","nonce":2523933697},{"type":"v1","addr":"172.21.15.53:6841","nonce":2523933697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/947644014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":947644014},{"type":"v1","addr":"172.21.15.104:6833","nonce":947644014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/3920144890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":3920144890},{"type":"v1","addr":"172.21.15.70:6833","nonce":3920144890}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:23:37.894565+0000","modified":"2023-07-14T19:25:00.337872+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24491,"mds_2":24461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/1013468175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":1013468175},{"type":"v1","addr":"172.21.15.104:6835","nonce":1013468175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24461":{"gid":24461,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.70:6835/2384359054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":2384359054},{"type":"v1","addr":"172.21.15.70:6835","nonce":2384359054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24463":{"gid":24463,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/1255142015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":1255142015},{"type":"v1","addr":"172.21.15.104:6837","nonce":1255142015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24484":{"gid":24484,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6839/1650908884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1650908884},{"type":"v1","addr":"172.21.15.104:6839","nonce":1650908884}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24491":{"gid":24491,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6837/3730963239","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3730963239},{"type":"v1","addr":"172.21.15.70:6837","nonce":3730963239}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6839/2496341573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":2496341573},{"type":"v1","addr":"172.21.15.70:6839","nonce":2496341573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:25:03.865 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f32240a12a0 msgr2=0x7f32240a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.866 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f32240a12a0 0x7f32240a3750 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3220005fd0 tx=0x7f3220005d00 comp rx=0 tx=0).stop 2023-07-14T19:25:03.866 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 msgr2=0x7f3238195860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:03.866 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 0x7f3238195860 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f3228000c00 tx=0x7f3228009f90 comp rx=0 tx=0).stop 2023-07-14T19:25:03.866 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 shutdown_connections 2023-07-14T19:25:03.866 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f32240a12a0 0x7f32240a3750 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.867 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f32380ffe00 0x7f323819ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.867 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32380fe280 0x7f3238195860 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.867 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 --2- 172.21.15.53:0/1195054758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32380ff470 0x7f3238195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:03.867 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 >> 172.21.15.53:0/1195054758 conn(0x7f32380f98a0 msgr2=0x7f3238104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:03.867 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 shutdown_connections 2023-07-14T19:25:03.868 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:03.864+0000 7f323dfae700 1 -- 172.21.15.53:0/1195054758 wait complete. 2023-07-14T19:25:03.868 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 24 2023-07-14T19:25:03.877 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:25:04.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.002+0000 7ff1da4a8700 1 -- 172.21.15.53:0/700711977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d4100680 msgr2=0x7ff1d4106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.004 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.002+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/700711977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d4100680 0x7ff1d4106f40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00b0a0 tx=0x7ff1cc00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.002+0000 7ff1da4a8700 1 -- 172.21.15.53:0/700711977 shutdown_connections 2023-07-14T19:25:04.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.002+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/700711977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d4100680 0x7ff1d4106f40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.002+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/700711977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d40feb00 0x7ff1d40fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.005 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.002+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/700711977 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d40ffcf0 0x7ff1d4100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.002+0000 7ff1da4a8700 1 -- 172.21.15.53:0/700711977 >> 172.21.15.53:0/700711977 conn(0x7ff1d40fa100 msgr2=0x7ff1d40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:04.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.002+0000 7ff1da4a8700 1 -- 172.21.15.53:0/700711977 shutdown_connections 2023-07-14T19:25:04.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.002+0000 7ff1da4a8700 1 -- 172.21.15.53:0/700711977 wait complete. 2023-07-14T19:25:04.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1da4a8700 1 Processor -- start 2023-07-14T19:25:04.006 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1da4a8700 1 -- start start 2023-07-14T19:25:04.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1da4a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d40feb00 0x7ff1d4068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.009 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1da4a8700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d40ffcf0 0x7ff1d419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1da4a8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d4100680 0x7ff1d419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1da4a8700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d419f7a0 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1da4a8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d419f910 con 0x7ff1d4100680 2023-07-14T19:25:04.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1da4a8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d419fad0 con 0x7ff1d40feb00 2023-07-14T19:25:04.010 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d40ffcf0 0x7ff1d419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.011 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d40ffcf0 0x7ff1d419cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51654/0 (socket says 172.21.15.53:51654) 2023-07-14T19:25:04.011 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d37fe700 1 -- 172.21.15.53:0/1220907177 learned_addr learned my addr 172.21.15.53:0/1220907177 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:04.011 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d3fff700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d40feb00 0x7ff1d4068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.011 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d8a45700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d4100680 0x7ff1d419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.012 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d37fe700 1 -- 172.21.15.53:0/1220907177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d40feb00 msgr2=0x7ff1d4068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.012 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d37fe700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d40feb00 0x7ff1d4068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.012 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d37fe700 1 -- 172.21.15.53:0/1220907177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d4100680 msgr2=0x7ff1d419f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.012 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d37fe700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d4100680 0x7ff1d419f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.012 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.003+0000 7ff1d37fe700 1 -- 172.21.15.53:0/1220907177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc009d00 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.013 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.004+0000 7ff1d37fe700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d40ffcf0 0x7ff1d419cba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800d950 tx=0x7ff1c800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:04.013 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.004+0000 7ff1d17fa700 1 -- 172.21.15.53:0/1220907177 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8009940 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.013 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.004+0000 7ff1d17fa700 1 -- 172.21.15.53:0/1220907177 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1c8010460 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.013 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.004+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d419fdb0 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.013 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.004+0000 7ff1d17fa700 1 -- 172.21.15.53:0/1220907177 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c800f620 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.004+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d41a0280 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.005+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d4105ff0 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.006+0000 7ff1d17fa700 1 -- 172.21.15.53:0/1220907177 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff1c80108d0 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.006+0000 7ff1d17fa700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff1bc0a12a0 0x7ff1bc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.007+0000 7ff1d17fa700 1 -- 172.21.15.53:0/1220907177 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7ff1c80ccb00 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.014 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.007+0000 7ff1d3fff700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff1bc0a12a0 0x7ff1bc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.007+0000 7ff1d3fff700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff1bc0a12a0 0x7ff1bc0a3750 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff1d40ffb50 tx=0x7ff1c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:04.015 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.010+0000 7ff1d17fa700 1 -- 172.21.15.53:0/1220907177 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff1c8096120 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.213 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.212+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1d410a350 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.215 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.213+0000 7ff1d17fa700 1 -- 172.21.15.53:0/1220907177 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22459 (secure 0 0 0) 0x7ff1c8016090 con 0x7ff1d40ffcf0 2023-07-14T19:25:04.215 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-07-14T19:25:04.215 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"fsid":"c5147628-227a-11ee-9b33-001a4aab830c","created":"2023-07-14T19:16:47.794884+0000","modified":"2023-07-14T19:24:42.310475+0000","last_up_change":"2023-07-14T19:22:55.991778+0000","last_in_change":"2023-07-14T19:22:38.214240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:19:51.775369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:23:35.935538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:23:37.252372+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"14c120ad-b975-4c88-b14e-4f72a0d61a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6803","nonce":243717916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6805","nonce":243717916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6809","nonce":243717916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":243717916},{"type":"v1","addr":"172.21.15.53:6807","nonce":243717916}]},"public_addr":"172.21.15.53:6803/243717916","cluster_addr":"172.21.15.53:6805/243717916","heartbeat_back_addr":"172.21.15.53:6809/243717916","heartbeat_front_addr":"172.21.15.53:6807/243717916","state":["exists","up"]},{"osd":1,"uuid":"301f0944-3cad-4319-9242-4a7e4c4bcc94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6801","nonce":1213220726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6803","nonce":1213220726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6807","nonce":1213220726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1213220726},{"type":"v1","addr":"172.21.15.70:6805","nonce":1213220726}]},"public_addr":"172.21.15.70:6801/1213220726","cluster_addr":"172.21.15.70:6803/1213220726","heartbeat_back_addr":"172.21.15.70:6807/1213220726","heartbeat_front_addr":"172.21.15.70:6805/1213220726","state":["exists","up"]},{"osd":2,"uuid":"7a6ae12c-bcf5-4ae1-98fb-bdec5fbe7447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6801","nonce":1379504827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6803","nonce":1379504827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6807","nonce":1379504827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":1379504827},{"type":"v1","addr":"172.21.15.104:6805","nonce":1379504827}]},"public_addr":"172.21.15.104:6801/1379504827","cluster_addr":"172.21.15.104:6803/1379504827","heartbeat_back_addr":"172.21.15.104:6807/1379504827","heartbeat_front_addr":"172.21.15.104:6805/1379504827","state":["exists","up"]},{"osd":3,"uuid":"4b06af11-cdd7-426e-8f3c-68b2b33e750a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6811","nonce":1549618174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6813","nonce":1549618174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6817","nonce":1549618174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1549618174},{"type":"v1","addr":"172.21.15.53:6815","nonce":1549618174}]},"public_addr":"172.21.15.53:6811/1549618174","cluster_addr":"172.21.15.53:6813/1549618174","heartbeat_back_addr":"172.21.15.53:6817/1549618174","heartbeat_front_addr":"172.21.15.53:6815/1549618174","state":["exists","up"]},{"osd":4,"uuid":"b7dd3e40-bc97-4894-92ec-21c756a018ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6809","nonce":195242658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6811","nonce":195242658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6815","nonce":195242658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":195242658},{"type":"v1","addr":"172.21.15.70:6813","nonce":195242658}]},"public_addr":"172.21.15.70:6809/195242658","cluster_addr":"172.21.15.70:6811/195242658","heartbeat_back_addr":"172.21.15.70:6815/195242658","heartbeat_front_addr":"172.21.15.70:6813/195242658","state":["exists","up"]},{"osd":5,"uuid":"530ff111-a77f-4eee-894f-0098aa1febd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6809","nonce":4101602275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6811","nonce":4101602275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6815","nonce":4101602275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":4101602275},{"type":"v1","addr":"172.21.15.104:6813","nonce":4101602275}]},"public_addr":"172.21.15.104:6809/4101602275","cluster_addr":"172.21.15.104:6811/4101602275","heartbeat_back_addr":"172.21.15.104:6815/4101602275","heartbeat_front_addr":"172.21.15.104:6813/4101602275","state":["exists","up"]},{"osd":6,"uuid":"b7ab09b5-1dfc-4d44-a248-7aac58e27749","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6819","nonce":4148952318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6821","nonce":4148952318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6825","nonce":4148952318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":4148952318},{"type":"v1","addr":"172.21.15.53:6823","nonce":4148952318}]},"public_addr":"172.21.15.53:6819/4148952318","cluster_addr":"172.21.15.53:6821/4148952318","heartbeat_back_addr":"172.21.15.53:6825/4148952318","heartbeat_front_addr":"172.21.15.53:6823/4148952318","state":["exists","up"]},{"osd":7,"uuid":"3b18c85a-6628-4f12-b07f-d292fcd3a05b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6817","nonce":2311839228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6819","nonce":2311839228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6823","nonce":2311839228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2311839228},{"type":"v1","addr":"172.21.15.70:6821","nonce":2311839228}]},"public_addr":"172.21.15.70:6817/2311839228","cluster_addr":"172.21.15.70:6819/2311839228","heartbeat_back_addr":"172.21.15.70:6823/2311839228","heartbeat_front_addr":"172.21.15.70:6821/2311839228","state":["exists","up"]},{"osd":8,"uuid":"521e5c70-e2f7-45f9-81fc-93bc6401c6b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6817","nonce":48256695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6819","nonce":48256695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6823","nonce":48256695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":48256695},{"type":"v1","addr":"172.21.15.104:6821","nonce":48256695}]},"public_addr":"172.21.15.104:6817/48256695","cluster_addr":"172.21.15.104:6819/48256695","heartbeat_back_addr":"172.21.15.104:6823/48256695","heartbeat_front_addr":"172.21.15.104:6821/48256695","state":["exists","up"]},{"osd":9,"uuid":"c5795443-41f4-4230-9a33-9795ebaffe91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6827","nonce":3378335665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6829","nonce":3378335665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6833","nonce":3378335665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":3378335665},{"type":"v1","addr":"172.21.15.53:6831","nonce":3378335665}]},"public_addr":"172.21.15.53:6827/3378335665","cluster_addr":"172.21.15.53:6829/3378335665","heartbeat_back_addr":"172.21.15.53:6833/3378335665","heartbeat_front_addr":"172.21.15.53:6831/3378335665","state":["exists","up"]},{"osd":10,"uuid":"ff27354b-283d-406f-b2ea-fe1c9f769e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6825","nonce":1946322503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6827","nonce":1946322503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6831","nonce":1946322503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1946322503},{"type":"v1","addr":"172.21.15.70:6829","nonce":1946322503}]},"public_addr":"172.21.15.70:6825/1946322503","cluster_addr":"172.21.15.70:6827/1946322503","heartbeat_back_addr":"172.21.15.70:6831/1946322503","heartbeat_front_addr":"172.21.15.70:6829/1946322503","state":["exists","up"]},{"osd":11,"uuid":"c2d7cdf0-c8ac-4fc1-a017-e641c666619a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6825","nonce":3513976129}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6827","nonce":3513976129}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6831","nonce":3513976129}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3513976129},{"type":"v1","addr":"172.21.15.104:6829","nonce":3513976129}]},"public_addr":"172.21.15.104:6825/3513976129","cluster_addr":"172.21.15.104:6827/3513976129","heartbeat_back_addr":"172.21.15.104:6831/3513976129","heartbeat_front_addr":"172.21.15.104:6829/3513976129","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:05.780033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:26.902213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:19:48.198985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:09.295038+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:29.635931+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:20:50.760981+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:11.170112+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:31.666679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:21:52.456631+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:13.249974+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:33.416520+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:22:54.111644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/1700081881":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6801/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/2149831":"2023-07-15T19:17:41.240106+0000","172.21.15.53:0/4078421180":"2023-07-15T19:17:41.240106+0000","172.21.15.53:6800/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:6801/3434712469":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/228763847":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3718395334":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/3412815128":"2023-07-15T19:17:15.286956+0000","172.21.15.53:0/2206633343":"2023-07-15T19:17:41.240106+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:25:04.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff1bc0a12a0 msgr2=0x7ff1bc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.218 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff1bc0a12a0 0x7ff1bc0a3750 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff1d40ffb50 tx=0x7ff1c4009260 comp rx=0 tx=0).stop 2023-07-14T19:25:04.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d40ffcf0 msgr2=0x7ff1d419cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.219 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d40ffcf0 0x7ff1d419cba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800d950 tx=0x7ff1c800dc60 comp rx=0 tx=0).stop 2023-07-14T19:25:04.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 shutdown_connections 2023-07-14T19:25:04.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7ff1bc0a12a0 0x7ff1bc0a3750 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.220 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d4100680 0x7ff1d419f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff1d40feb00 0x7ff1d4068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.221 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 --2- 172.21.15.53:0/1220907177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1d40ffcf0 0x7ff1d419cba0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 >> 172.21.15.53:0/1220907177 conn(0x7ff1d40fa100 msgr2=0x7ff1d4103130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:04.222 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 shutdown_connections 2023-07-14T19:25:04.223 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.216+0000 7ff1da4a8700 1 -- 172.21.15.53:0/1220907177 wait complete. 2023-07-14T19:25:04.230 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-07-14T19:25:04.359 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.358+0000 7f040bda7700 1 -- 172.21.15.53:0/1891435090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040feea0 msgr2=0x7f04040ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.360 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.358+0000 7f040bda7700 1 --2- 172.21.15.53:0/1891435090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040feea0 0x7f04040ff310 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009990 tx=0x7f03f4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.360 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.358+0000 7f040bda7700 1 -- 172.21.15.53:0/1891435090 shutdown_connections 2023-07-14T19:25:04.360 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.358+0000 7f040bda7700 1 --2- 172.21.15.53:0/1891435090 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04040ff850 0x7f04040ffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.361 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.358+0000 7f040bda7700 1 --2- 172.21.15.53:0/1891435090 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04040fe550 0x7f04040fe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.361 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.358+0000 7f040bda7700 1 --2- 172.21.15.53:0/1891435090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040feea0 0x7f04040ff310 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.361 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.358+0000 7f040bda7700 1 -- 172.21.15.53:0/1891435090 >> 172.21.15.53:0/1891435090 conn(0x7f04040fa1a0 msgr2=0x7f04040fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:04.361 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.358+0000 7f040bda7700 1 -- 172.21.15.53:0/1891435090 shutdown_connections 2023-07-14T19:25:04.362 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.358+0000 7f040bda7700 1 -- 172.21.15.53:0/1891435090 wait complete. 2023-07-14T19:25:04.367 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040bda7700 1 Processor -- start 2023-07-14T19:25:04.367 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040bda7700 1 -- start start 2023-07-14T19:25:04.368 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040bda7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04040fe550 0x7f0404197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.368 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040bda7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04040feea0 0x7f0404198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.369 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040bda7700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040ff850 0x7f040419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.369 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040bda7700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040419f660 con 0x7f04040ff850 2023-07-14T19:25:04.369 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040bda7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040419f7a0 con 0x7f04040feea0 2023-07-14T19:25:04.370 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040bda7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040419f930 con 0x7f04040fe550 2023-07-14T19:25:04.370 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040a344700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040ff850 0x7f040419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.370 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040a344700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040ff850 0x7f040419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51670/0 (socket says 172.21.15.53:51670) 2023-07-14T19:25:04.371 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f040a344700 1 -- 172.21.15.53:0/170147344 learned_addr learned my addr 172.21.15.53:0/170147344 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:04.371 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f0409342700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04040feea0 0x7f0404198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.359+0000 7f0409b43700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04040fe550 0x7f0404197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f040a344700 1 -- 172.21.15.53:0/170147344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04040fe550 msgr2=0x7f0404197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.372 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f040a344700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04040fe550 0x7f0404197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f040a344700 1 -- 172.21.15.53:0/170147344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04040feea0 msgr2=0x7f0404198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f040a344700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04040feea0 0x7f0404198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f040a344700 1 -- 172.21.15.53:0/170147344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f40095f0 con 0x7f04040ff850 2023-07-14T19:25:04.373 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f040a344700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040ff850 0x7f040419f090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f03fc007ee0 tx=0x7f03fc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:04.374 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f03faffd700 1 -- 172.21.15.53:0/170147344 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc017070 con 0x7f04040ff850 2023-07-14T19:25:04.374 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f03faffd700 1 -- 172.21.15.53:0/170147344 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03fc00f040 con 0x7f04040ff850 2023-07-14T19:25:04.374 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040419fc10 con 0x7f04040ff850 2023-07-14T19:25:04.374 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f03faffd700 1 -- 172.21.15.53:0/170147344 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc005580 con 0x7f04040ff850 2023-07-14T19:25:04.374 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.360+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04041a00d0 con 0x7f04040ff850 2023-07-14T19:25:04.375 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.361+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0404109330 con 0x7f04040ff850 2023-07-14T19:25:04.375 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.362+0000 7f03faffd700 1 -- 172.21.15.53:0/170147344 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f03fc00db20 con 0x7f04040ff850 2023-07-14T19:25:04.375 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.363+0000 7f03faffd700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f03f00a1260 0x7f03f00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.375 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.363+0000 7f0409b43700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f03f00a1260 0x7f03f00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.375 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.363+0000 7f03faffd700 1 -- 172.21.15.53:0/170147344 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8190+0+0 (secure 0 0 0) 0x7f03fc0cc230 con 0x7f04040ff850 2023-07-14T19:25:04.376 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.363+0000 7f0409b43700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f03f00a1260 0x7f03f00a3710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0400009c60 tx=0x7f0400009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:04.376 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.367+0000 7f03faffd700 1 -- 172.21.15.53:0/170147344 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f03fc0957f0 con 0x7f04040ff850 2023-07-14T19:25:04.571 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:04 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/814153170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:04 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1195054758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.572 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:04 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1220907177' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.617 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.616+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f040402cac0 con 0x7f04040ff850 2023-07-14T19:25:04.622 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.621+0000 7f03faffd700 1 -- 172.21.15.53:0/170147344 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7f03fc098e10 con 0x7f04040ff850 2023-07-14T19:25:04.624 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f03f00a1260 msgr2=0x7f03f00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.625 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f03f00a1260 0x7f03f00a3710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0400009c60 tx=0x7f0400009260 comp rx=0 tx=0).stop 2023-07-14T19:25:04.625 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040ff850 msgr2=0x7f040419f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.625 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040ff850 0x7f040419f090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f03fc007ee0 tx=0x7f03fc00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 shutdown_connections 2023-07-14T19:25:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f03f00a1260 0x7f03f00a3710 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f04040ff850 0x7f040419f090 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f04040fe550 0x7f0404197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 --2- 172.21.15.53:0/170147344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04040feea0 0x7f0404198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 >> 172.21.15.53:0/170147344 conn(0x7f04040fa1a0 msgr2=0x7f04040fbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:04.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 shutdown_connections 2023-07-14T19:25:04.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.623+0000 7f040bda7700 1 -- 172.21.15.53:0/170147344 wait complete. 2023-07-14T19:25:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-07-14T19:25:04.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-07-14T19:25:04.653 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/814153170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.654 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1195054758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.654 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1220907177' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.655 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/814153170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.655 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1195054758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.656 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1220907177' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.769 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.768+0000 7f6c70964700 1 -- 172.21.15.53:0/2622853287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 msgr2=0x7f6c6c1080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.768+0000 7f6c70964700 1 --2- 172.21.15.53:0/2622853287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 0x7f6c6c1080c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400b0a0 tx=0x7f6c6400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.768+0000 7f6c70964700 1 -- 172.21.15.53:0/2622853287 shutdown_connections 2023-07-14T19:25:04.770 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.768+0000 7f6c70964700 1 --2- 172.21.15.53:0/2622853287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 0x7f6c6c1080c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.768+0000 7f6c70964700 1 --2- 172.21.15.53:0/2622853287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6c0ffc80 0x7f6c6c100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.768+0000 7f6c70964700 1 --2- 172.21.15.53:0/2622853287 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c6c100e70 0x7f6c6c1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.768+0000 7f6c70964700 1 -- 172.21.15.53:0/2622853287 >> 172.21.15.53:0/2622853287 conn(0x7f6c6c0fb2a0 msgr2=0x7f6c6c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:04.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.768+0000 7f6c70964700 1 -- 172.21.15.53:0/2622853287 shutdown_connections 2023-07-14T19:25:04.772 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.768+0000 7f6c70964700 1 -- 172.21.15.53:0/2622853287 wait complete. 2023-07-14T19:25:04.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c70964700 1 Processor -- start 2023-07-14T19:25:04.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c70964700 1 -- start start 2023-07-14T19:25:04.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c70964700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6c0ffc80 0x7f6c6c195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c70964700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c6c100e70 0x7f6c6c195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.774 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c70964700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 0x7f6c6c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c70964700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c19d5a0 con 0x7f6c6c101800 2023-07-14T19:25:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c70964700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c19d6e0 con 0x7f6c6c0ffc80 2023-07-14T19:25:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c70964700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c19d870 con 0x7f6c6c100e70 2023-07-14T19:25:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c6ad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 0x7f6c6c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c6ad9d700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 0x7f6c6c19cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51684/0 (socket says 172.21.15.53:51684) 2023-07-14T19:25:04.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c6ad9d700 1 -- 172.21.15.53:0/1848787686 learned_addr learned my addr 172.21.15.53:0/1848787686 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:04.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c69d9b700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c6c100e70 0x7f6c6c195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c6a59c700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6c0ffc80 0x7f6c6c195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.776 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c6ad9d700 1 -- 172.21.15.53:0/1848787686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c6c100e70 msgr2=0x7f6c6c195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c6ad9d700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c6c100e70 0x7f6c6c195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c6ad9d700 1 -- 172.21.15.53:0/1848787686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6c0ffc80 msgr2=0x7f6c6c195a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c6ad9d700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6c0ffc80 0x7f6c6c195a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c6ad9d700 1 -- 172.21.15.53:0/1848787686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c64009d00 con 0x7f6c6c101800 2023-07-14T19:25:04.777 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.769+0000 7f6c69d9b700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c6c100e70 0x7f6c6c195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:25:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.770+0000 7f6c6ad9d700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 0x7f6c6c19cfd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400b780 tx=0x7f6c6400bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.770+0000 7f6c5b7fe700 1 -- 172.21.15.53:0/1848787686 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6400e040 con 0x7f6c6c101800 2023-07-14T19:25:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.770+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c19daf0 con 0x7f6c6c101800 2023-07-14T19:25:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.770+0000 7f6c5b7fe700 1 -- 172.21.15.53:0/1848787686 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c64004e60 con 0x7f6c6c101800 2023-07-14T19:25:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.770+0000 7f6c5b7fe700 1 -- 172.21.15.53:0/1848787686 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6401b870 con 0x7f6c6c101800 2023-07-14T19:25:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.770+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c19dfc0 con 0x7f6c6c101800 2023-07-14T19:25:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.771+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6c1071a0 con 0x7f6c6c101800 2023-07-14T19:25:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.772+0000 7f6c5b7fe700 1 -- 172.21.15.53:0/1848787686 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6c6401b9d0 con 0x7f6c6c101800 2023-07-14T19:25:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.772+0000 7f6c5b7fe700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6c540a11e0 0x7f6c540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.773+0000 7f6c5b7fe700 1 -- 172.21.15.53:0/1848787686 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8448+0+0 (secure 0 0 0) 0x7f6c640cea10 con 0x7f6c6c101800 2023-07-14T19:25:04.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.773+0000 7f6c6a59c700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6c540a11e0 0x7f6c540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:04.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.773+0000 7f6c6a59c700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6c540a11e0 0x7f6c540a3690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c100cd0 tx=0x7f6c60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:04.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:04.776+0000 7f6c5b7fe700 1 -- 172.21.15.53:0/1848787686 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6c64097fc0 con 0x7f6c6c101800 2023-07-14T19:25:04.978 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-b[123413]: ignoring --setuser ceph since I am not root 2023-07-14T19:25:04.978 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-b[123413]: ignoring --setgroup ceph since I am not root 2023-07-14T19:25:04.978 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-mds[123437]: -- 172.21.15.70:0/3000485185 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e8b09b51e0 con 0x55e8b098ac00 2023-07-14T19:25:04.979 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-b[123413]: starting mds.b at 2023-07-14T19:25:04.979 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-e[122724]: ignoring --setuser ceph since I am not root 2023-07-14T19:25:04.979 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-e[122724]: ignoring --setgroup ceph since I am not root 2023-07-14T19:25:04.980 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-mds[122748]: -- 172.21.15.70:0/3285980853 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5611cf92d1e0 con 0x5611cf902c00 2023-07-14T19:25:04.980 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-e[122724]: starting mds.e at 2023-07-14T19:25:04.980 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-h[123074]: ignoring --setuser ceph since I am not root 2023-07-14T19:25:04.981 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-h[123074]: ignoring --setgroup ceph since I am not root 2023-07-14T19:25:04.981 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-mds[123098]: -- 172.21.15.70:0/3708762269 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5578f71111e0 con 0x5578f70e6c00 2023-07-14T19:25:04.981 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-h[123074]: starting mds.h at 2023-07-14T19:25:04.981 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:25:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-h[123074]: 2023-07-14T19:25:04.657+0000 7fb4a7f5a700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-07-14T19:25:05.010 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-f[124252]: ignoring --setuser ceph since I am not root 2023-07-14T19:25:05.011 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-f[124252]: ignoring --setgroup ceph since I am not root 2023-07-14T19:25:05.011 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-mds[124276]: -- 172.21.15.104:0/4118898186 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563cbd0591e0 con 0x563cbd02ec00 2023-07-14T19:25:05.011 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-f[124252]: starting mds.f at 2023-07-14T19:25:05.012 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-c[124596]: ignoring --setuser ceph since I am not root 2023-07-14T19:25:05.012 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-c[124596]: ignoring --setgroup ceph since I am not root 2023-07-14T19:25:05.012 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-mds[124620]: -- 172.21.15.104:0/3221737283 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55abb85fd1e0 con 0x55abb85d3000 2023-07-14T19:25:05.012 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-c[124596]: starting mds.c at 2023-07-14T19:25:05.013 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-i[124944]: ignoring --setuser ceph since I am not root 2023-07-14T19:25:05.013 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-i[124944]: ignoring --setgroup ceph since I am not root 2023-07-14T19:25:05.013 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-mds[124968]: -- 172.21.15.104:0/2172529308 <== mon.0 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e34c5bb1e0 con 0x55e34c591000 2023-07-14T19:25:05.013 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:25:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-i[124944]: starting mds.i at 2023-07-14T19:25:05.022 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.021+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6c6c02cd30 con 0x7f6c6c101800 2023-07-14T19:25:05.071 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:04 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-a[96541]: 2023-07-14T19:25:04.616+0000 7fb8ac602700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:25:05.627 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.625+0000 7f6c5b7fe700 1 -- 172.21.15.53:0/1848787686 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6c6401bc80 con 0x7f6c6c101800 2023-07-14T19:25:05.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6c540a11e0 msgr2=0x7f6c540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:05.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6c540a11e0 0x7f6c540a3690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c100cd0 tx=0x7f6c60009260 comp rx=0 tx=0).stop 2023-07-14T19:25:05.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 msgr2=0x7f6c6c19cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:05.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 0x7f6c6c19cfd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400b780 tx=0x7f6c6400bdb0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 shutdown_connections 2023-07-14T19:25:05.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f6c540a11e0 0x7f6c540a3690 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c6c101800 0x7f6c6c19cfd0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6c0ffc80 0x7f6c6c195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 --2- 172.21.15.53:0/1848787686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c6c100e70 0x7f6c6c195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.631 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 >> 172.21.15.53:0/1848787686 conn(0x7f6c6c0fb2a0 msgr2=0x7f6c6c1042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:05.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 shutdown_connections 2023-07-14T19:25:05.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.628+0000 7f6c70964700 1 -- 172.21.15.53:0/1848787686 wait complete. 2023-07-14T19:25:05.643 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:05 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/170147344' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:25:05.643 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:05 smithi053 ceph-mon[96565]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:25:05.644 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:05 smithi053 ceph-mon[96565]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:25:05.644 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:05 smithi053 ceph-mon[96565]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:25:05.644 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:05 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/170147344' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:25:05.644 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:05 smithi053 ceph-mon[96565]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T19:25:05.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:05 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1848787686' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:25:05.645 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:05 smithi053 ceph-mon[96565]: pgmap v295: 97 pgs: 97 active+clean; 643 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-07-14T19:25:05.646 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-07-14T19:25:05.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:05 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/170147344' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:25:05.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:05 smithi070 ceph-mon[98015]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:25:05.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:05 smithi070 ceph-mon[98015]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:25:05.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:05 smithi070 ceph-mon[98015]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:25:05.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:05 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/170147344' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:25:05.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:05 smithi070 ceph-mon[98015]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T19:25:05.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:05 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1848787686' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:25:05.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:05 smithi070 ceph-mon[98015]: pgmap v295: 97 pgs: 97 active+clean; 643 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-07-14T19:25:05.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:05 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/170147344' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:25:05.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:05 smithi104 ceph-mon[98435]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:25:05.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:05 smithi104 ceph-mon[98435]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:25:05.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:05 smithi104 ceph-mon[98435]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:25:05.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:05 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/170147344' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:25:05.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:05 smithi104 ceph-mon[98435]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T19:25:05.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:05 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1848787686' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:25:05.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:05 smithi104 ceph-mon[98435]: pgmap v295: 97 pgs: 97 active+clean; 643 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-07-14T19:25:05.778 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.777+0000 7f862e028700 1 -- 172.21.15.53:0/208562040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8628100e30 msgr2=0x7f8628101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:05.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.777+0000 7f862e028700 1 --2- 172.21.15.53:0/208562040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8628100e30 0x7f8628101280 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f861c009990 tx=0x7f861c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.777+0000 7f862e028700 1 -- 172.21.15.53:0/208562040 shutdown_connections 2023-07-14T19:25:05.779 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.777+0000 7f862e028700 1 --2- 172.21.15.53:0/208562040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86281017c0 0x7f8628108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.777+0000 7f862e028700 1 --2- 172.21.15.53:0/208562040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86280ffc40 0x7f8628100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.777+0000 7f862e028700 1 --2- 172.21.15.53:0/208562040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8628100e30 0x7f8628101280 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.777+0000 7f862e028700 1 -- 172.21.15.53:0/208562040 >> 172.21.15.53:0/208562040 conn(0x7f86280fb260 msgr2=0x7f86280fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:05.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.777+0000 7f862e028700 1 -- 172.21.15.53:0/208562040 shutdown_connections 2023-07-14T19:25:05.780 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.777+0000 7f862e028700 1 -- 172.21.15.53:0/208562040 wait complete. 2023-07-14T19:25:05.782 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f862e028700 1 Processor -- start 2023-07-14T19:25:05.783 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f862e028700 1 -- start start 2023-07-14T19:25:05.783 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f862e028700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86280ffc40 0x7f86280743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:05.783 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f862e028700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8628100e30 0x7f8628072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:05.783 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f862e028700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86281017c0 0x7f8628072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f862e028700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8628073710 con 0x7f86280ffc40 2023-07-14T19:25:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f862e028700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8628073880 con 0x7f8628100e30 2023-07-14T19:25:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f862e028700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862819fa40 con 0x7f86281017c0 2023-07-14T19:25:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f86277fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86280ffc40 0x7f86280743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f86277fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86280ffc40 0x7f86280743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51694/0 (socket says 172.21.15.53:51694) 2023-07-14T19:25:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f86277fe700 1 -- 172.21.15.53:0/1404082357 learned_addr learned my addr 172.21.15.53:0/1404082357 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f8626ffd700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8628100e30 0x7f8628072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f8627fff700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86281017c0 0x7f8628072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f86277fe700 1 -- 172.21.15.53:0/1404082357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86281017c0 msgr2=0x7f8628072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:05.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f86277fe700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86281017c0 0x7f8628072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f86277fe700 1 -- 172.21.15.53:0/1404082357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8628100e30 msgr2=0x7f8628072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:05.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f86277fe700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8628100e30 0x7f8628072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:05.786 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.778+0000 7f86277fe700 1 -- 172.21.15.53:0/1404082357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c0095f0 con 0x7f86280ffc40 2023-07-14T19:25:05.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.779+0000 7f86277fe700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86280ffc40 0x7f86280743f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8620007f10 tx=0x7f862000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:05.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.779+0000 7f8624ff9700 1 -- 172.21.15.53:0/1404082357 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620017070 con 0x7f86280ffc40 2023-07-14T19:25:05.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.779+0000 7f8624ff9700 1 -- 172.21.15.53:0/1404082357 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f862000f040 con 0x7f86280ffc40 2023-07-14T19:25:05.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.779+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862819fd20 con 0x7f86280ffc40 2023-07-14T19:25:05.787 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.779+0000 7f8624ff9700 1 -- 172.21.15.53:0/1404082357 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620004630 con 0x7f86280ffc40 2023-07-14T19:25:05.788 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.779+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86281a0180 con 0x7f86280ffc40 2023-07-14T19:25:05.788 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.780+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86281071d0 con 0x7f86280ffc40 2023-07-14T19:25:05.788 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.781+0000 7f8624ff9700 1 -- 172.21.15.53:0/1404082357 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8620004790 con 0x7f86280ffc40 2023-07-14T19:25:05.788 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.782+0000 7f8624ff9700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86100a11e0 0x7f86100a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:05.789 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.782+0000 7f8626ffd700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86100a11e0 0x7f86100a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:05.789 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.782+0000 7f8624ff9700 1 -- 172.21.15.53:0/1404082357 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8448+0+0 (secure 0 0 0) 0x7f86200cc2a0 con 0x7f86280ffc40 2023-07-14T19:25:05.790 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.782+0000 7f8626ffd700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86100a11e0 0x7f86100a3690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f861c00b5c0 tx=0x7f861c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:05.790 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.786+0000 7f8624ff9700 1 -- 172.21.15.53:0/1404082357 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f86200957c0 con 0x7f86280ffc40 2023-07-14T19:25:05.986 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:05.985+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f862802c8e0 con 0x7f86280ffc40 2023-07-14T19:25:06.629 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.627+0000 7f8624ff9700 1 -- 172.21.15.53:0/1404082357 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v84) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8620098de0 con 0x7f86280ffc40 2023-07-14T19:25:06.632 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.630+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86100a11e0 msgr2=0x7f86100a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:06.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.630+0000 7f862e028700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86100a11e0 0x7f86100a3690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f861c00b5c0 tx=0x7f861c005960 comp rx=0 tx=0).stop 2023-07-14T19:25:06.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.630+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86280ffc40 msgr2=0x7f86280743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:06.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.630+0000 7f862e028700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86280ffc40 0x7f86280743f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8620007f10 tx=0x7f862000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.630+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 shutdown_connections 2023-07-14T19:25:06.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.630+0000 7f862e028700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f86100a11e0 0x7f86100a3690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.630+0000 7f862e028700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86281017c0 0x7f8628072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.630+0000 7f862e028700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86280ffc40 0x7f86280743f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.630+0000 7f862e028700 1 --2- 172.21.15.53:0/1404082357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8628100e30 0x7f8628072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.631+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 >> 172.21.15.53:0/1404082357 conn(0x7f86280fb260 msgr2=0x7f8628104310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.631+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 shutdown_connections 2023-07-14T19:25:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.631+0000 7f862e028700 1 -- 172.21.15.53:0/1404082357 wait complete. 2023-07-14T19:25:06.636 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' removed 2023-07-14T19:25:06.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:25:06.646 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:25:06.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: Cluster is now healthy 2023-07-14T19:25:06.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6835/4178383686,v1:172.21.15.70:6837/4178383686] up:boot 2023-07-14T19:25:06.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6838/1329870151,v1:172.21.15.70:6839/1329870151] up:boot 2023-07-14T19:25:06.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.70:6834/242520794,v1:172.21.15.70:6836/242520794] up:boot 2023-07-14T19:25:06.647 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.104:6834/4029163201,v1:172.21.15.104:6835/4029163201] up:boot 2023-07-14T19:25:06.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.104:6836/2609909928,v1:172.21.15.104:6837/2609909928] up:boot 2023-07-14T19:25:06.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: mds.? [v2:172.21.15.104:6838/367258509,v1:172.21.15.104:6839/367258509] up:boot 2023-07-14T19:25:06.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1848787686' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:25:06.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: fsmap 12 up:standby 2023-07-14T19:25:06.648 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:25:06.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:25:06.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:25:06.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:25:06.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:25:06.649 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:25:06.650 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:06 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1404082357' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:25:06.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-07-14T19:25:06.728 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:25:06.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:25:06.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: Cluster is now healthy 2023-07-14T19:25:06.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6835/4178383686,v1:172.21.15.70:6837/4178383686] up:boot 2023-07-14T19:25:06.729 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6838/1329870151,v1:172.21.15.70:6839/1329870151] up:boot 2023-07-14T19:25:06.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.70:6834/242520794,v1:172.21.15.70:6836/242520794] up:boot 2023-07-14T19:25:06.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.104:6834/4029163201,v1:172.21.15.104:6835/4029163201] up:boot 2023-07-14T19:25:06.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.104:6836/2609909928,v1:172.21.15.104:6837/2609909928] up:boot 2023-07-14T19:25:06.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: mds.? [v2:172.21.15.104:6838/367258509,v1:172.21.15.104:6839/367258509] up:boot 2023-07-14T19:25:06.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1848787686' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:25:06.730 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: fsmap 12 up:standby 2023-07-14T19:25:06.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:25:06.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:25:06.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:25:06.731 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:25:06.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:25:06.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:25:06.732 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:06 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1404082357' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:25:06.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:25:06.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:25:06.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: Cluster is now healthy 2023-07-14T19:25:06.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6835/4178383686,v1:172.21.15.70:6837/4178383686] up:boot 2023-07-14T19:25:06.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6838/1329870151,v1:172.21.15.70:6839/1329870151] up:boot 2023-07-14T19:25:06.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.70:6834/242520794,v1:172.21.15.70:6836/242520794] up:boot 2023-07-14T19:25:06.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.104:6834/4029163201,v1:172.21.15.104:6835/4029163201] up:boot 2023-07-14T19:25:06.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.104:6836/2609909928,v1:172.21.15.104:6837/2609909928] up:boot 2023-07-14T19:25:06.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: mds.? [v2:172.21.15.104:6838/367258509,v1:172.21.15.104:6839/367258509] up:boot 2023-07-14T19:25:06.762 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1848787686' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:25:06.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: fsmap 12 up:standby 2023-07-14T19:25:06.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:25:06.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:25:06.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:25:06.763 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:25:06.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:25:06.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: from='mgr.14152 172.21.15.53:0/3201950680' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:25:06.764 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:06 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1404082357' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:25:06.788 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1842810851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4103070 msgr2=0x7f4dc4105450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1842810851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4103070 0x7f4dc4105450 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009990 tx=0x7f4db4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1842810851 shutdown_connections 2023-07-14T19:25:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1842810851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dc4105990 0x7f4dc4107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1842810851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc4100750 0x7f4dc4102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1842810851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4103070 0x7f4dc4105450 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1842810851 >> 172.21.15.53:0/1842810851 conn(0x7f4dc40fa130 msgr2=0x7f4dc40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1842810851 shutdown_connections 2023-07-14T19:25:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1842810851 wait complete. 2023-07-14T19:25:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 Processor -- start 2023-07-14T19:25:06.792 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.787+0000 7f4dc9ce2700 1 -- start start 2023-07-14T19:25:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc9ce2700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4100750 0x7f4dc4195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc9ce2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dc4103070 0x7f4dc4195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc9ce2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc4105990 0x7f4dc419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:06.793 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc9ce2700 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc419d4d0 con 0x7f4dc4100750 2023-07-14T19:25:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc9ce2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc419d610 con 0x7f4dc4103070 2023-07-14T19:25:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc9ce2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc419d7a0 con 0x7f4dc4105990 2023-07-14T19:25:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4100750 0x7f4dc4195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc37fe700 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4100750 0x7f4dc4195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51708/0 (socket says 172.21.15.53:51708) 2023-07-14T19:25:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc37fe700 1 -- 172.21.15.53:0/1251040119 learned_addr learned my addr 172.21.15.53:0/1251040119 (peer_addr_for_me v2:172.21.15.53:0/0) 2023-07-14T19:25:06.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc2ffd700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dc4103070 0x7f4dc4195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:06.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc37fe700 1 -- 172.21.15.53:0/1251040119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc4105990 msgr2=0x7f4dc419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:06.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc3fff700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc4105990 0x7f4dc419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:06.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc37fe700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc4105990 0x7f4dc419cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.795 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc37fe700 1 -- 172.21.15.53:0/1251040119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dc4103070 msgr2=0x7f4dc4195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:06.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc37fe700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dc4103070 0x7f4dc4195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:06.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc37fe700 1 -- 172.21.15.53:0/1251040119 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db40095f0 con 0x7f4dc4100750 2023-07-14T19:25:06.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc2ffd700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dc4103070 0x7f4dc4195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:25:06.796 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc37fe700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4100750 0x7f4dc4195950 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00d980 tx=0x7f4dac00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:06.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc0ff9700 1 -- 172.21.15.53:0/1251040119 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac009b40 con 0x7f4dc4100750 2023-07-14T19:25:06.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc0ff9700 1 -- 172.21.15.53:0/1251040119 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4dac00de60 con 0x7f4dc4100750 2023-07-14T19:25:06.797 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc419da80 con 0x7f4dc4100750 2023-07-14T19:25:06.798 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc0ff9700 1 -- 172.21.15.53:0/1251040119 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac005580 con 0x7f4dc4100750 2023-07-14T19:25:06.798 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.788+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc419df90 con 0x7f4dc4100750 2023-07-14T19:25:06.798 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.789+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc410b430 con 0x7f4dc4100750 2023-07-14T19:25:06.798 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.791+0000 7f4dc0ff9700 1 -- 172.21.15.53:0/1251040119 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4dac004750 con 0x7f4dc4100750 2023-07-14T19:25:06.798 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.791+0000 7f4dc0ff9700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4db00a1260 0x7f4db00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:06.799 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.791+0000 7f4dc2ffd700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4db00a1260 0x7f4db00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:06.799 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.791+0000 7f4dc0ff9700 1 -- 172.21.15.53:0/1251040119 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7915+0+0 (secure 0 0 0) 0x7f4dac0ccc50 con 0x7f4dc4100750 2023-07-14T19:25:06.799 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.792+0000 7f4dc2ffd700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4db00a1260 0x7f4db00a3710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4db400b5c0 tx=0x7f4db4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:06.800 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.794+0000 7f4dc0ff9700 1 -- 172.21.15.53:0/1251040119 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4dac0964b0 con 0x7f4dc4100750 2023-07-14T19:25:06.992 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:06.991+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4dc410b5c0 con 0x7f4dc4100750 2023-07-14T19:25:07.630 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.629+0000 7f4dc0ff9700 1 -- 172.21.15.53:0/1251040119 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4dac017030 con 0x7f4dc4100750 2023-07-14T19:25:07.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4db00a1260 msgr2=0x7f4db00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:07.633 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4db00a1260 0x7f4db00a3710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4db400b5c0 tx=0x7f4db4005fb0 comp rx=0 tx=0).stop 2023-07-14T19:25:07.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4100750 msgr2=0x7f4dc4195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:07.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4100750 0x7f4dc4195950 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00d980 tx=0x7f4dac00dd40 comp rx=0 tx=0).stop 2023-07-14T19:25:07.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 shutdown_connections 2023-07-14T19:25:07.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:6800/3014344459,v1:172.21.15.53:6801/3014344459] conn(0x7f4db00a1260 0x7f4db00a3710 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:07.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4dc4105990 0x7f4dc419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:07.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc4100750 0x7f4dc4195950 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:07.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 --2- 172.21.15.53:0/1251040119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dc4103070 0x7f4dc4195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:07.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.631+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 >> 172.21.15.53:0/1251040119 conn(0x7f4dc40fa130 msgr2=0x7f4dc4101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:07.635 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.632+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 shutdown_connections 2023-07-14T19:25:07.636 INFO:teuthology.orchestra.run.smithi053.stderr:2023-07-14T19:25:07.632+0000 7f4dc9ce2700 1 -- 172.21.15.53:0/1251040119 wait complete. 2023-07-14T19:25:07.636 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' removed 2023-07-14T19:25:07.647 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:25:07.677 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:25:07.713 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:25:07.750 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-14T19:25:07.751 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:25:07.770 DEBUG:teuthology.orchestra.run.smithi070:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:25:07.788 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:25:07.808 INFO:tasks.cephadm:Stopping all daemons... 2023-07-14T19:25:07.809 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-14T19:25:07.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.a 2023-07-14T19:25:07.977 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:07 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1404082357' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:25:07.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:07 smithi070 ceph-mon[98015]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:25:07.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:07 smithi070 ceph-mon[98015]: from='client.? 172.21.15.53:0/1251040119' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:25:07.978 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:07 smithi070 ceph-mon[98015]: pgmap v297: 65 pgs: 65 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:25:08.010 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:07 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1404082357' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:25:08.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:07 smithi104 ceph-mon[98435]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:25:08.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:07 smithi104 ceph-mon[98435]: from='client.? 172.21.15.53:0/1251040119' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:25:08.011 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:07 smithi104 ceph-mon[98435]: pgmap v297: 65 pgs: 65 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:25:08.071 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:07 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1404082357' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:25:08.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:07 smithi053 ceph-mon[96565]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:25:08.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:07 smithi053 ceph-mon[96565]: from='client.? 172.21.15.53:0/1251040119' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:25:08.072 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:07 smithi053 ceph-mon[96565]: pgmap v297: 65 pgs: 65 active+clean; 577 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:25:08.073 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:07 smithi053 systemd[1]: Stopping Ceph mon.a for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:08.466 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-a[96541]: 2023-07-14T19:25:08.173+0000 7fb8b1e0d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:08.466 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-a[96541]: 2023-07-14T19:25:08.173+0000 7fb8b1e0d700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-14T19:25:08.821 INFO:journalctl@ceph.mon.a.smithi053.stdout:Jul 14 19:25:08 smithi053 bash[145445]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-a 2023-07-14T19:25:08.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.a.service' 2023-07-14T19:25:08.938 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:08.939 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-14T19:25:08.940 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-14T19:25:08.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.b 2023-07-14T19:25:09.227 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:08 smithi070 systemd[1]: Stopping Ceph mon.b for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:09.663 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-b[97991]: 2023-07-14T19:25:09.354+0000 7fdc9c6e4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:09.663 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-b[97991]: 2023-07-14T19:25:09.354+0000 7fdc9c6e4700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-14T19:25:09.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Jul 14 19:25:09 smithi070 bash[125179]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-b 2023-07-14T19:25:10.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.b.service' 2023-07-14T19:25:10.083 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:10.084 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-14T19:25:10.084 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-14T19:25:10.085 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.c 2023-07-14T19:25:10.444 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:10 smithi104 systemd[1]: Stopping Ceph mon.c for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:10.760 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:10 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-c[98398]: 2023-07-14T19:25:10.441+0000 7f7dec653700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:10.761 INFO:journalctl@ceph.mon.c.smithi104.stdout:Jul 14 19:25:10 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mon-c[98398]: 2023-07-14T19:25:10.441+0000 7f7dec653700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-14T19:25:11.089 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mon.c.service' 2023-07-14T19:25:11.169 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:11.170 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-14T19:25:11.170 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-14T19:25:11.171 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T19:25:11.517 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:25:11 smithi053 systemd[1]: Stopping Ceph mgr.x for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:11.775 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:25:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x[96810]: 2023-07-14T19:25:11.514+0000 7f4e27f4d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:11.776 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Jul 14 19:25:11 smithi053 bash[145965]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-x 2023-07-14T19:25:11.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.x.service' 2023-07-14T19:25:12.008 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:12.009 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-14T19:25:12.010 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-14T19:25:12.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.y 2023-07-14T19:25:12.422 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:25:12 smithi070 systemd[1]: Stopping Ceph mgr.y for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:12.712 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:25:12 smithi070 bash[125521]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-y 2023-07-14T19:25:12.893 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Jul 14 19:25:12 smithi070 systemd[1]: ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-14T19:25:12.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.y.service' 2023-07-14T19:25:12.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:12.937 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-14T19:25:12.937 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-14T19:25:12.937 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.z 2023-07-14T19:25:13.260 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:25:12 smithi104 systemd[1]: Stopping Ceph mgr.z for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:13.645 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:25:13 smithi104 bash[127072]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mgr-z 2023-07-14T19:25:13.826 INFO:journalctl@ceph.mgr.z.smithi104.stdout:Jul 14 19:25:13 smithi104 systemd[1]: ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-14T19:25:13.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mgr.z.service' 2023-07-14T19:25:13.867 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:13.867 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-14T19:25:13.868 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-14T19:25:13.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.0 2023-07-14T19:25:14.215 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:25:13 smithi053 systemd[1]: Stopping Ceph osd.0 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:14.571 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:25:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-0[109751]: 2023-07-14T19:25:14.213+0000 7fa289f36700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:14.572 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:25:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-0[109751]: 2023-07-14T19:25:14.213+0000 7fa289f36700 -1 osd.0 85 *** Got signal Terminated *** 2023-07-14T19:25:14.572 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:25:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-0[109751]: 2023-07-14T19:25:14.213+0000 7fa289f36700 -1 osd.0 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:25:19.908 INFO:journalctl@ceph.osd.0.smithi053.stdout:Jul 14 19:25:19 smithi053 bash[146109]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-0 2023-07-14T19:25:21.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.0.service' 2023-07-14T19:25:21.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:21.121 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-14T19:25:21.121 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-14T19:25:21.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.1 2023-07-14T19:25:21.478 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:25:21 smithi070 systemd[1]: Stopping Ceph osd.1 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:21.738 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:25:21 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-1[103115]: 2023-07-14T19:25:21.485+0000 7efd6b6da700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:21.739 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:25:21 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-1[103115]: 2023-07-14T19:25:21.485+0000 7efd6b6da700 -1 osd.1 85 *** Got signal Terminated *** 2023-07-14T19:25:21.739 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:25:21 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-1[103115]: 2023-07-14T19:25:21.485+0000 7efd6b6da700 -1 osd.1 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:25:27.157 INFO:journalctl@ceph.osd.1.smithi070.stdout:Jul 14 19:25:27 smithi070 bash[125673]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-1 2023-07-14T19:25:28.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.1.service' 2023-07-14T19:25:28.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:28.472 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-14T19:25:28.472 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-14T19:25:28.473 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.2 2023-07-14T19:25:28.834 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:25:28 smithi104 systemd[1]: Stopping Ceph osd.2 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:29.260 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:25:28 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-2[104321]: 2023-07-14T19:25:28.831+0000 7f9d25f82700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:29.261 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:25:28 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-2[104321]: 2023-07-14T19:25:28.831+0000 7f9d25f82700 -1 osd.2 85 *** Got signal Terminated *** 2023-07-14T19:25:29.261 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:25:28 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-2[104321]: 2023-07-14T19:25:28.831+0000 7f9d25f82700 -1 osd.2 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:25:34.510 INFO:journalctl@ceph.osd.2.smithi104.stdout:Jul 14 19:25:34 smithi104 bash[127223]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-2 2023-07-14T19:25:35.689 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.2.service' 2023-07-14T19:25:35.730 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:35.732 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-14T19:25:35.733 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-14T19:25:35.733 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.3 2023-07-14T19:25:36.072 INFO:journalctl@ceph.osd.3.smithi053.stdout:Jul 14 19:25:35 smithi053 systemd[1]: Stopping Ceph osd.3 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:36.572 INFO:journalctl@ceph.osd.3.smithi053.stdout:Jul 14 19:25:36 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-3[115173]: 2023-07-14T19:25:36.094+0000 7f0d1a82e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:36.573 INFO:journalctl@ceph.osd.3.smithi053.stdout:Jul 14 19:25:36 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-3[115173]: 2023-07-14T19:25:36.094+0000 7f0d1a82e700 -1 osd.3 85 *** Got signal Terminated *** 2023-07-14T19:25:36.573 INFO:journalctl@ceph.osd.3.smithi053.stdout:Jul 14 19:25:36 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-3[115173]: 2023-07-14T19:25:36.094+0000 7f0d1a82e700 -1 osd.3 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:25:41.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:41 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:41.050+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:42.071 INFO:journalctl@ceph.osd.3.smithi053.stdout:Jul 14 19:25:41 smithi053 bash[146378]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-3 2023-07-14T19:25:42.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:42 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:42.088+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:43.047 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:42 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:42.988+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:43.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.3.service' 2023-07-14T19:25:43.097 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:43.097 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-14T19:25:43.098 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-14T19:25:43.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.4 2023-07-14T19:25:43.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:42 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:42.914+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:43.465 INFO:journalctl@ceph.osd.4.smithi070.stdout:Jul 14 19:25:43 smithi070 systemd[1]: Stopping Ceph osd.4 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:43.465 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:43 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:43.051+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:43.727 INFO:journalctl@ceph.osd.4.smithi070.stdout:Jul 14 19:25:43 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-4[108565]: 2023-07-14T19:25:43.463+0000 7f1d80924700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:43.728 INFO:journalctl@ceph.osd.4.smithi070.stdout:Jul 14 19:25:43 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-4[108565]: 2023-07-14T19:25:43.463+0000 7f1d80924700 -1 osd.4 85 *** Got signal Terminated *** 2023-07-14T19:25:43.728 INFO:journalctl@ceph.osd.4.smithi070.stdout:Jul 14 19:25:43 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-4[108565]: 2023-07-14T19:25:43.463+0000 7f1d80924700 -1 osd.4 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:25:43.760 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:43 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:43.276+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:44.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:43 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:43.948+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:44.289 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:43 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:43.960+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:44.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:44 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:44.094+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:44.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:44 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:44.288+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:44.760 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:44 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:44.263+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:45.248 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:44 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:44.907+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:45.321 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:44 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:44.961+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:45.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:45 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:45.056+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:45.510 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:45 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:45.247+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:45.821 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:45 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:45.332+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:46.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:45 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:45.887+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:46.321 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:45 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:45.913+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:46.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:46 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:46.093+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:46.760 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:46 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:46.291+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:46.821 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:46 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:46.359+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:47.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:46 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:46.881+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:47.321 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:46 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:46.910+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:47.443 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:47 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:47.128+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:47.727 INFO:journalctl@ceph.osd.4.smithi070.stdout:Jul 14 19:25:47 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-4[108565]: 2023-07-14T19:25:47.442+0000 7f1d7a716700 -1 osd.4 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.272505+0000 front 2023-07-14T19:25:25.272532+0000 (oldest deadline 2023-07-14T19:25:46.972782+0000) 2023-07-14T19:25:47.760 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:47 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:47.296+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:47.761 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:47 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:47.296+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.062013+0000 front 2023-07-14T19:25:23.061960+0000 (oldest deadline 2023-07-14T19:25:47.162019+0000) 2023-07-14T19:25:47.821 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:47 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:47.338+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:48.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:47 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:47.925+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:48.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:47 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:47.925+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:48.306 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:47 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:47.919+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:48.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:48 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:48.101+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:48.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:48 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:48.101+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:48.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:48 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:48.305+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:48.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:48 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:48.305+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:48.760 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:48 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:48.271+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:48.761 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:48 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:48.272+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.062013+0000 front 2023-07-14T19:25:23.061960+0000 (oldest deadline 2023-07-14T19:25:47.162019+0000) 2023-07-14T19:25:49.227 INFO:journalctl@ceph.osd.4.smithi070.stdout:Jul 14 19:25:49 smithi070 bash[125936]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-4 2023-07-14T19:25:49.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:49 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:49.077+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:49.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:49 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:49.077+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:49.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:48 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:48.952+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:49.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:48 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:48.952+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:49.269 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:48 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:48.965+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:49.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:49 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:49.268+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:49.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:49 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:49.268+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:49.728 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:49 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:49.305+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:49.760 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:49.283+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:49.761 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:49.283+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.062013+0000 front 2023-07-14T19:25:23.061960+0000 (oldest deadline 2023-07-14T19:25:47.162019+0000) 2023-07-14T19:25:50.244 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:49 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:49.964+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:50.252 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:50 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:50.073+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:50.252 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:49.908+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:50.253 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:49 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:49.908+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:50.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:50 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:50.287+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:50.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:50 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:50.051+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:50.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:50 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:50.051+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:50.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.4.service' 2023-07-14T19:25:50.510 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:50 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:50.250+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:50.511 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:50 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:50.250+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.062013+0000 front 2023-07-14T19:25:23.061960+0000 (oldest deadline 2023-07-14T19:25:47.162019+0000) 2023-07-14T19:25:50.542 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:50.542 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-14T19:25:50.543 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-14T19:25:50.543 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.5 2023-07-14T19:25:50.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:50 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:50.243+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:50.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:50 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:50.243+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:50.895 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:50 smithi104 systemd[1]: Stopping Ceph osd.5 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:51.239 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:51 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:51.037+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:51.239 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:50 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:50.894+0000 7f6bc745b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:51.240 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:50 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:50.894+0000 7f6bc745b700 -1 osd.5 85 *** Got signal Terminated *** 2023-07-14T19:25:51.240 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:50 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:50.894+0000 7f6bc745b700 -1 osd.5 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:25:51.240 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:51 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:51.238+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:51.240 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:50 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:50.905+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:51.241 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:50 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:50.905+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:51.275 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:51 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:51.004+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:51.275 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:51 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:51.004+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:25:51.477 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:51 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:51.268+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:51.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:51 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:51.039+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:51.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:51 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:51.039+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:51.512 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:51 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:51.238+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.062013+0000 front 2023-07-14T19:25:23.061960+0000 (oldest deadline 2023-07-14T19:25:47.162019+0000) 2023-07-14T19:25:51.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:51 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:51.274+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:51.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:51 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:51.274+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:52.244 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:52 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:52.014+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:52.244 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:51 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:51.872+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:52.244 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:51 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:51.872+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:52.257 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:51 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:51.996+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:52.258 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:51 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:51.996+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:52.321 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:52 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:52.028+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:52.322 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:52 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:52.028+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:25:52.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:52 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:52.248+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:52.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:52 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:52.248+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:52.510 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:52 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:52.242+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:52.511 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:52 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:52.242+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.062013+0000 front 2023-07-14T19:25:23.061960+0000 (oldest deadline 2023-07-14T19:25:47.162019+0000) 2023-07-14T19:25:52.727 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:52 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:52.256+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:53.208 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:53 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:53.042+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:53.208 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:52 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:52.917+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:53.208 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:52 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:52.917+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:53.230 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:52 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:52.976+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:53.230 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:52 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:52.976+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:53.263 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:53 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:53.011+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:53.263 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:53 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:53.011+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:25:53.510 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:53 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:53.207+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:53.510 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:53 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:53.207+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.062013+0000 front 2023-07-14T19:25:23.061960+0000 (oldest deadline 2023-07-14T19:25:47.162019+0000) 2023-07-14T19:25:53.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:53 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:53.262+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:53.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:53 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:53.262+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:53.727 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:53 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:53.229+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:54.192 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:54 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:54.043+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:54.192 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:53 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:53.909+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:54.193 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:53 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:53.909+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:54.227 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:53 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:53.930+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:54.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:53 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:53.930+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:54.321 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:54 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:54.052+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:54.322 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:54 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:54.052+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:25:54.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:54 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:54.241+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:54.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:54 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:54.241+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:54.510 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:54 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:54.191+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:54.511 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:54 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:54.191+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.062013+0000 front 2023-07-14T19:25:23.061960+0000 (oldest deadline 2023-07-14T19:25:47.162019+0000) 2023-07-14T19:25:54.511 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:54 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:54.191+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:30.063054+0000 front 2023-07-14T19:25:30.063030+0000 (oldest deadline 2023-07-14T19:25:54.163206+0000) 2023-07-14T19:25:54.727 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:54 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:54.246+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:55.225 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:54 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:54.894+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:55.226 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:54 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:54.894+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:55.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:55 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:55.006+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:55.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:54 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:54.944+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:55.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:54 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:54.944+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:55.261 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:55 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:55.147+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.161212+0000 front 2023-07-14T19:25:17.161180+0000 (oldest deadline 2023-07-14T19:25:43.061366+0000) 2023-07-14T19:25:55.262 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:55 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:55.147+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.062013+0000 front 2023-07-14T19:25:23.061960+0000 (oldest deadline 2023-07-14T19:25:47.162019+0000) 2023-07-14T19:25:55.262 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:55 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5[109793]: 2023-07-14T19:25:55.147+0000 7f6bc0a4c700 -1 osd.5 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:30.063054+0000 front 2023-07-14T19:25:30.063030+0000 (oldest deadline 2023-07-14T19:25:54.163206+0000) 2023-07-14T19:25:55.477 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:55 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:55.224+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:55.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:55 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:55.224+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:25:55.571 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:55 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:55.089+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:55.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:55 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:55.089+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:25:55.573 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:55 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:55.089+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.201290+0000 front 2023-07-14T19:25:31.201422+0000 (oldest deadline 2023-07-14T19:25:54.602083+0000) 2023-07-14T19:25:55.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:55 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:55.193+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:55.574 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:55 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:55.193+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:56.209 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:55 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:55.919+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:56.210 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:55 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:55.919+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:56.210 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:55 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:55.919+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:25:56.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:55 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:55.994+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:56.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:55 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:55.994+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:25:56.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:55 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:55.958+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:56.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:55 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:55.958+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:56.477 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:56 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:56.208+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:56.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:56 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:56.208+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:25:56.571 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:56.075+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:56.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:56.075+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:25:56.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:56.075+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.201290+0000 front 2023-07-14T19:25:31.201422+0000 (oldest deadline 2023-07-14T19:25:54.602083+0000) 2023-07-14T19:25:56.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:56.172+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:56.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:56.172+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:56.760 INFO:journalctl@ceph.osd.5.smithi104.stdout:Jul 14 19:25:56 smithi104 bash[127480]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-5 2023-07-14T19:25:57.227 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:56 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:56.939+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:57.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:56 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:56.939+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:57.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:56 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:56.939+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:25:57.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:56 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:56.995+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:57.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:56 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:56.995+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:25:57.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:56 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:56.971+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:57.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:56 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:56.971+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:57.322 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:57 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:57.027+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:57.322 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:57 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:57.027+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:25:57.323 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:57 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:57.027+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.201290+0000 front 2023-07-14T19:25:31.201422+0000 (oldest deadline 2023-07-14T19:25:54.602083+0000) 2023-07-14T19:25:57.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:57 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:57.134+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:57.324 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:57 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:57.134+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:57.727 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:57 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:57.226+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:57.728 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:57 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:57.226+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:25:57.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.5.service' 2023-07-14T19:25:57.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:25:57.927 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-14T19:25:57.927 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-14T19:25:57.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.6 2023-07-14T19:25:58.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:57 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:57.992+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:58.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:57 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:57.992+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:25:58.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:57 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:57.988+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:58.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:57 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:57.988+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:58.292 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:58.094+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:58.293 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:58.094+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:58.294 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:57 smithi053 systemd[1]: Stopping Ceph osd.6 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:25:58.294 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:58.053+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:58.294 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:58.053+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:25:58.295 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:58.053+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.201290+0000 front 2023-07-14T19:25:31.201422+0000 (oldest deadline 2023-07-14T19:25:54.602083+0000) 2023-07-14T19:25:58.295 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:58.291+0000 7f00bd939700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:25:58.477 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:58 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:58.223+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:58.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:58 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:58.223+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:25:58.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:57 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:57.979+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:58.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:57 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:57.979+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:58.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:57 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:57.979+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:25:58.571 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:58.291+0000 7f00bd939700 -1 osd.6 85 *** Got signal Terminated *** 2023-07-14T19:25:58.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:58 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:58.291+0000 7f00bd939700 -1 osd.6 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:25:59.227 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:58 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:58.932+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:25:59.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:58 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:58.932+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:25:59.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:58 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:58.932+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:25:59.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:59 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:59.003+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:25:59.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:25:59 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:25:59.003+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:25:59.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:59 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:59.025+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:25:59.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:59 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:59.025+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:25:59.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:25:59 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:25:59.025+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:25:59.321 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:59 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:59.053+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:25:59.322 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:59 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:59.053+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:25:59.322 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:25:59 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:25:59.053+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.201290+0000 front 2023-07-14T19:25:31.201422+0000 (oldest deadline 2023-07-14T19:25:54.602083+0000) 2023-07-14T19:25:59.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:59 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:59.127+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:25:59.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:59 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:59.127+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:25:59.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:25:59 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:25:59.127+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:25:59.727 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:59 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:59.226+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:25:59.728 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:25:59 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:25:59.226+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:00.477 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:00 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:00.183+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:00.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:00 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:00.183+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:00.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:59 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:59.980+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:00.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:59 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:59.980+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:00.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:25:59 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:25:59.980+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:00.510 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:00 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:00.018+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:00.511 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:00 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:00.018+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:00.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:00 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:00.051+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:00.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:00 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:00.051+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:00.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:00 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:00.051+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:00.571 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:00.095+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:26:00.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:00.095+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:26:00.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:00.095+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.201290+0000 front 2023-07-14T19:25:31.201422+0000 (oldest deadline 2023-07-14T19:25:54.602083+0000) 2023-07-14T19:26:00.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:00.169+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:00.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:00.169+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:00.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:00 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:00.169+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:01.228 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:01 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:01.153+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:01.229 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:01 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:01.153+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:01.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:00 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:00.951+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:01.230 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:00 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:00.951+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:01.230 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:00 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:00.951+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:01.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:01 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:01.009+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:01.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:01 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:01.009+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:01.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:01 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:01.052+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:01.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:01 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:01.052+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:01.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:01 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:01.052+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:01.571 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:01 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:01.107+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:26:01.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:01 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:01.107+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:26:01.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:01 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:01.107+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.201290+0000 front 2023-07-14T19:25:31.201422+0000 (oldest deadline 2023-07-14T19:25:54.602083+0000) 2023-07-14T19:26:01.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:01 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:01.120+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:01.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:01 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:01.120+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:01.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:01 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:01.120+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:02.227 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:02 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:02.176+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:02.228 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:02 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:02.176+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:02.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:01 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:01.933+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:02.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:01 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:01.933+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:02.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:01 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:01.933+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:02.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:01 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:01.933+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:02.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:01 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:01.961+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:02.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:01 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:01.961+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:02.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:02 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:02.033+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:02.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:02 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:02.033+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:02.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:02 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:02.033+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:02.571 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:02 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:02.124+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:26:02.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:02 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:02.124+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:26:02.572 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:02 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:02.124+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.201290+0000 front 2023-07-14T19:25:31.201422+0000 (oldest deadline 2023-07-14T19:25:54.602083+0000) 2023-07-14T19:26:02.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:02 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:02.143+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:02.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:02 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:02.143+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:02.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:02 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:02.143+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:03.227 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:03 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:03.128+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:03.228 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:03 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:03.128+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:03.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:02 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:02.937+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:03.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:02 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:02.937+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:03.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:02 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:02.937+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:03.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:02 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:02.937+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:03.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:02 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:02.974+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:03.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:02 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:02.974+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:03.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:02 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:02.974+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:03.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:03 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:03.011+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:03.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:03 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:03.011+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:03.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:03 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:03.011+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:03.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:03 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:03.139+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:03.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:03 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:03.139+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:03.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:03 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:03.139+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:03.573 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:03 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:03.105+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.998362+0000 front 2023-07-14T19:25:18.998407+0000 (oldest deadline 2023-07-14T19:25:42.498678+0000) 2023-07-14T19:26:03.574 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:03 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:03.105+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.499370+0000 front 2023-07-14T19:25:22.499386+0000 (oldest deadline 2023-07-14T19:25:50.700054+0000) 2023-07-14T19:26:03.574 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:03 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:03.105+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.201290+0000 front 2023-07-14T19:25:31.201422+0000 (oldest deadline 2023-07-14T19:25:54.602083+0000) 2023-07-14T19:26:03.575 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:03 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6[120627]: 2023-07-14T19:26:03.105+0000 7f00b9f30700 -1 osd.6 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:38.702766+0000 front 2023-07-14T19:25:38.702836+0000 (oldest deadline 2023-07-14T19:26:02.802789+0000) 2023-07-14T19:26:04.162 INFO:journalctl@ceph.osd.6.smithi053.stdout:Jul 14 19:26:04 smithi053 bash[146650]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-6 2023-07-14T19:26:04.227 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:04.115+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:04.228 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:04.115+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:04.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:03 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:03.907+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:04.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:03 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:03.907+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:04.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:03 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:03.907+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:04.230 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:03 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:03.907+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:04.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:03 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:03.964+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:04.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:03 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:03.964+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:04.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:03 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:03.964+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:04.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:04.011+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:04.263 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:04.011+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:04.263 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:04.011+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:04.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:04 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:04.160+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:04.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:04 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:04.160+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:04.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:04 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:04.160+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:04.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:04 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:04.160+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:05.227 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:04.951+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:05.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:04.951+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:05.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:04.951+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:05.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:04 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:04.951+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:05.228 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:05.145+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:05.229 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:05.145+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:05.229 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:05.145+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.776087+0000 front 2023-07-14T19:25:40.776159+0000 (oldest deadline 2023-07-14T19:26:04.276030+0000) 2023-07-14T19:26:05.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:04.975+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:05.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:04.975+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:05.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:04 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:04.975+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:05.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:05 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:05.017+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:05.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:05 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:05.017+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:05.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:05 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:05.017+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:05.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:05 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:05.145+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:05.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:05 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:05.145+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:05.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:05 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:05.145+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:05.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:05 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:05.145+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:05.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.6.service' 2023-07-14T19:26:05.409 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:26:05.411 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-14T19:26:05.411 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-14T19:26:05.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.7 2023-07-14T19:26:05.727 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:05 smithi070 systemd[1]: Stopping Ceph osd.7 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:26:06.132 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:05.784+0000 7f3ced070700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:26:06.132 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:05.784+0000 7f3ced070700 -1 osd.7 85 *** Got signal Terminated *** 2023-07-14T19:26:06.133 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:05.784+0000 7f3ced070700 -1 osd.7 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:26:06.133 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:06 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:06.131+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:06.133 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:06 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:06.131+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:06.133 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:05.974+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:06.134 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:05.974+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:06.134 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:05.974+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:06.134 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:05.974+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:06.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:05 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:05.942+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:06.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:05 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:05.942+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:06.262 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:05 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:05.942+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:06.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:06 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:06.047+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:06.263 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:06 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:06.047+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:06.263 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:06 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:06.047+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:06.477 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:06 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:06.131+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.776087+0000 front 2023-07-14T19:25:40.776159+0000 (oldest deadline 2023-07-14T19:26:04.276030+0000) 2023-07-14T19:26:06.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:06 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:06.118+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:06.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:06 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:06.118+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:06.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:06 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:06.118+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:06.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:06 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:06.118+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:07.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:06 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:06.933+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:07.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:06 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:06.933+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:07.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:06 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:06.933+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:07.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:07 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:07.014+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:07.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:07 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:07.014+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:07.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:07 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:07.014+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:07.477 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:07 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:07.169+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:07.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:07 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:07.169+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:07.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:07 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:07.169+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.776087+0000 front 2023-07-14T19:25:40.776159+0000 (oldest deadline 2023-07-14T19:26:04.276030+0000) 2023-07-14T19:26:07.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:07 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:07.011+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:07.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:07 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:07.011+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:07.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:07 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:07.011+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:07.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:07 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:07.011+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:07.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:07.144+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:07.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:07.144+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:07.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:07.144+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:07.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:07 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:07.144+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:08.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:07 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:07.899+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:08.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:07 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:07.899+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:08.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:07 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:07.899+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:08.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:08 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:08.045+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:08.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:08 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:08.045+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:08.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:08 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:08.045+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:08.477 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:08 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:08.212+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:08.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:08 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:08.212+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:08.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:08 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:08.212+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.776087+0000 front 2023-07-14T19:25:40.776159+0000 (oldest deadline 2023-07-14T19:26:04.276030+0000) 2023-07-14T19:26:08.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:08 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:07.999+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:08.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:08 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:07.999+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:08.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:08 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:07.999+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:08.480 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:08 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:07.999+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:08.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:08.166+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:08.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:08.166+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:08.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:08.166+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:08.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:08 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:08.166+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:09.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:08 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:08.877+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:09.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:08 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:08.877+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:09.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:08 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:08.877+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:09.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:09 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:09.085+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:09.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:09 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:09.085+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:09.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:09 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:09.085+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:09.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:09.177+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:09.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:09.177+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:09.479 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:09.177+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.776087+0000 front 2023-07-14T19:25:40.776159+0000 (oldest deadline 2023-07-14T19:26:04.276030+0000) 2023-07-14T19:26:09.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:09.046+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:09.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:09.046+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:09.480 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:09.046+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:09.480 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:09.046+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:09.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:09 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:09.192+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:09.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:09 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:09.192+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:09.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:09 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:09.192+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:09.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:09 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:09.192+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:10.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:09 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:09.920+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:10.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:09 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:09.920+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:10.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:09 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:09.920+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:10.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:09 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:09.920+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:44.441139+0000 front 2023-07-14T19:25:44.441019+0000 (oldest deadline 2023-07-14T19:26:09.641414+0000) 2023-07-14T19:26:10.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:10 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:10.082+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:10.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:10 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:10.082+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:10.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:10 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:10.082+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:10.477 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:10 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:10.185+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.172387+0000 front 2023-07-14T19:25:25.172340+0000 (oldest deadline 2023-07-14T19:25:49.272568+0000) 2023-07-14T19:26:10.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:10 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:10.185+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:31.474538+0000 front 2023-07-14T19:25:31.474786+0000 (oldest deadline 2023-07-14T19:25:54.374562+0000) 2023-07-14T19:26:10.478 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:10 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7[114094]: 2023-07-14T19:26:10.185+0000 7f3ce6e62700 -1 osd.7 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.776087+0000 front 2023-07-14T19:25:40.776159+0000 (oldest deadline 2023-07-14T19:26:04.276030+0000) 2023-07-14T19:26:10.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:10 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:10.033+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:10.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:10 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:10.033+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:10.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:10 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:10.033+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:10.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:10 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:10.033+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:10.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:10 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:10.194+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:10.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:10 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:10.194+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:10.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:10 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:10.194+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:10.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:10 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:10.194+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:11.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:10 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:10.957+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:11.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:10 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:10.957+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:11.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:10 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:10.957+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:11.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:10 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:10.957+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:44.441139+0000 front 2023-07-14T19:25:44.441019+0000 (oldest deadline 2023-07-14T19:26:09.641414+0000) 2023-07-14T19:26:11.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:11 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:11.099+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:11.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:11 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:11.099+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:11.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:11 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:11.099+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:11.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:11 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:11.099+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:11.313 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:11 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:11.027+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:11.314 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:11 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:11.027+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:11.314 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:11 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:11.027+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:11.315 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:11 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:11.027+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:11.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:11.152+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:11.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:11.152+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:11.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:11.152+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:11.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:11 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:11.152+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:11.977 INFO:journalctl@ceph.osd.7.smithi070.stdout:Jul 14 19:26:11 smithi070 bash[126198]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-7 2023-07-14T19:26:12.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:11 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:11.919+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:12.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:11 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:11.919+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:12.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:11 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:11.919+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:12.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:11 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:11.919+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:44.441139+0000 front 2023-07-14T19:25:44.441019+0000 (oldest deadline 2023-07-14T19:26:09.641414+0000) 2023-07-14T19:26:12.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:12 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:12.141+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:12.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:12 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:12.141+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:12.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:12 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:12.141+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:12.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:12 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:12.141+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:12.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:12 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:12.032+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:12.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:12 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:12.032+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:12.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:12 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:12.032+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:12.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:12 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:12.032+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:12.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:12.105+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:12.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:12.105+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:12.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:12.105+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:12.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:12.105+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:12.574 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:12 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:12.105+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:12.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.7.service' 2023-07-14T19:26:12.958 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:26:12.959 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-14T19:26:12.959 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-14T19:26:12.960 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.8 2023-07-14T19:26:13.227 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:13 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:13.072+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:13.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:13 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:13.072+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:13.228 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:13 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:13.072+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:13.229 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:13 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:13.072+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:13.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:13.097+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:13.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:13.097+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:13.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:13.097+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:13.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:13.097+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:13.262 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:12 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:12.963+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:13.262 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:12 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:12.963+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:13.263 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:12 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:12.963+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:13.263 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:12 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:12.963+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:44.441139+0000 front 2023-07-14T19:25:44.441019+0000 (oldest deadline 2023-07-14T19:26:09.641414+0000) 2023-07-14T19:26:13.263 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:12 smithi104 systemd[1]: Stopping Ceph osd.8 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:26:13.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:13.088+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:13.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:13.088+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:13.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:13.088+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:13.572 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:13.088+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:13.573 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:13 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:13.088+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:13.760 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:13.339+0000 7f9db27e4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:26:13.760 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:13.339+0000 7f9db27e4700 -1 osd.8 85 *** Got signal Terminated *** 2023-07-14T19:26:13.761 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:13.339+0000 7f9db27e4700 -1 osd.8 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:26:14.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:13.953+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:14.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:13.953+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:14.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:13.953+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:14.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:13 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:13.953+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:44.441139+0000 front 2023-07-14T19:25:44.441019+0000 (oldest deadline 2023-07-14T19:26:09.641414+0000) 2023-07-14T19:26:14.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:14 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:14.110+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:14.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:14 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:14.110+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:14.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:14 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:14.110+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:14.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:14 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:14.110+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:14.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:14.043+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:14.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:14.043+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:14.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:14.043+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:14.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:14.043+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:14.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:14 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:14.043+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:14.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:14 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:14.054+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:14.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:14 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:14.054+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:14.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:14 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:14.054+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:14.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:14 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:14.054+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:14.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:14 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:14.054+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:15.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:14 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:14.965+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:15.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:14 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:14.965+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:15.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:14 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:14.965+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:15.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:14 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:14.965+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:44.441139+0000 front 2023-07-14T19:25:44.441019+0000 (oldest deadline 2023-07-14T19:26:09.641414+0000) 2023-07-14T19:26:15.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:15 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:15.138+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:15.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:15 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:15.138+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:15.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:15 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:15.138+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:15.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:15 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:15.138+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:15.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.029+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:15.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.029+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:15.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.029+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:15.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.029+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:15.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.029+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:15.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:15 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:15.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:15.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:15 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:15.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:15.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:15 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:15.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:15.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:15 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:15.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:15.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:15 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:15.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:16.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:16.007+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:16.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:16.008+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:16.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:16.008+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:16.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:16.008+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:44.441139+0000 front 2023-07-14T19:25:44.441019+0000 (oldest deadline 2023-07-14T19:26:09.641414+0000) 2023-07-14T19:26:16.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:16.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:16.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:16.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:16.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:16.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:16.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:16.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:16.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.980+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:16.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.980+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:16.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.980+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:16.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.980+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:16.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:15 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:15.980+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:16.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:16 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:16.057+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:16.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:16 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:16.057+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:16.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:16 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:16.057+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:16.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:16 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:16.057+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:16.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:16 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:16.057+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:17.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:16.990+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:17.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:16.990+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:17.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:16.990+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:17.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:16 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:16.990+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:44.441139+0000 front 2023-07-14T19:25:44.441019+0000 (oldest deadline 2023-07-14T19:26:09.641414+0000) 2023-07-14T19:26:17.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:17 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:17.112+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:17.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:17 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:17.112+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:17.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:17 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:17.112+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:17.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:17 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:17.112+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:17.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:16 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:16.945+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:17.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:16 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:16.945+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:17.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:16 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:16.945+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:17.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:16 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:16.945+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:17.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:16 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:16.945+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:17.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:17 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:17.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:17.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:17 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:17.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:17.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:17 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:17.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:17.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:17 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:17.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:17.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:17 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:17.047+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:18.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:18.006+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:24.237225+0000 front 2023-07-14T19:25:24.237293+0000 (oldest deadline 2023-07-14T19:25:49.536814+0000) 2023-07-14T19:26:18.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:18.006+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.439145+0000 front 2023-07-14T19:25:33.439187+0000 (oldest deadline 2023-07-14T19:25:55.739334+0000) 2023-07-14T19:26:18.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:18.006+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.440280+0000 front 2023-07-14T19:25:37.440304+0000 (oldest deadline 2023-07-14T19:26:02.740187+0000) 2023-07-14T19:26:18.261 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8[115384]: 2023-07-14T19:26:18.006+0000 7f9dac5d6700 -1 osd.8 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:44.441139+0000 front 2023-07-14T19:25:44.441019+0000 (oldest deadline 2023-07-14T19:26:09.641414+0000) 2023-07-14T19:26:18.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:18.110+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:18.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:18.110+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:18.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:18.110+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:18.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:18.110+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:18.263 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:18.110+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:18.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:17 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:17.915+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:18.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:17 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:17.915+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:18.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:17 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:17.915+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:18.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:17 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:17.915+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:18.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:17 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:17.915+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:18.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:18 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:18.016+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:18.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:18 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:18.016+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:18.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:18 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:18.016+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:18.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:18 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:18.016+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:18.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:18 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:18.016+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:18.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:18 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:18.016+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:19.260 INFO:journalctl@ceph.osd.8.smithi104.stdout:Jul 14 19:26:19 smithi104 bash[127748]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-8 2023-07-14T19:26:19.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:19 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:19.149+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:19.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:19 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:19.149+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:19.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:19 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:19.149+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:19.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:19 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:19.149+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:19.262 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:19 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:19.149+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:19.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:18 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:18.889+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:19.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:18 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:18.889+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:19.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:18 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:18.889+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:19.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:18 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:18.889+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:19.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:18 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:18.889+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:19.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:19 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:19.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:19.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:19 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:19.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:19.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:19 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:19.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:19.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:19 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:19.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:19.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:19 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:19.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:19.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:19 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:19.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:20.260 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:20 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:20.173+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:20.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:20 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:20.173+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:20.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:20 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:20.173+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:20.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:20 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:20.173+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:20.261 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:20 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:20.173+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:20.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:19 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:19.898+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:20.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:19 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:19.898+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:20.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:19 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:19.898+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:20.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:19 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:19.898+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:20.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:19 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:19.898+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:20.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:20 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:20.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:20.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:20 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:20.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:20.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:20 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:20.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:20.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:20 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:20.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:20.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:20 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:20.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:20.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:20 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:20.044+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:20.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.8.service' 2023-07-14T19:26:20.714 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:26:20.715 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-14T19:26:20.716 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-14T19:26:20.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.9 2023-07-14T19:26:21.071 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:20 smithi053 systemd[1]: Stopping Ceph osd.9 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:26:21.072 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:20.882+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:21.072 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:20.883+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:21.072 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:20.883+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:21.073 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:20.883+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:21.073 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:20.883+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:21.073 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:20 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:20.883+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:51.343213+0000 front 2023-07-14T19:25:51.343122+0000 (oldest deadline 2023-07-14T19:26:20.143699+0000) 2023-07-14T19:26:21.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:21 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:21.066+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:21.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:21 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:21.067+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:21.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:21 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:21.067+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:21.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:21 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:21.067+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:21.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:21 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:21.067+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:21.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:21 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:21.067+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:21.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:21 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:21.208+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:21.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:21 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:21.208+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:21.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:21 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:21.208+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:21.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:21 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:21.208+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:21.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:21 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:21.208+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:21.571 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:21 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:21.072+0000 7f522b9ec700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:26:21.580 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:21 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:21.072+0000 7f522b9ec700 -1 osd.9 85 *** Got signal Terminated *** 2023-07-14T19:26:21.581 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:21 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:21.072+0000 7f522b9ec700 -1 osd.9 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:26:22.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:21 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:21.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:22.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:21 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:21.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:22.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:21 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:21.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:22.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:21 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:21.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:22.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:21 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:21.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:22.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:21 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:21.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:51.343213+0000 front 2023-07-14T19:25:51.343122+0000 (oldest deadline 2023-07-14T19:26:20.143699+0000) 2023-07-14T19:26:22.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:22 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:22.055+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:22.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:22 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:22.055+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:22.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:22 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:22.055+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:22.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:22 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:22.055+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:22.480 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:22 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:22.055+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:22.480 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:22 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:22.055+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:22.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:22 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:22.184+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:22.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:22 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:22.184+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:22.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:22 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:22.184+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:22.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:22 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:22.184+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:22.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:22 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:22.184+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:23.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:22 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:22.925+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:23.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:22 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:22.925+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:23.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:22 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:22.925+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:23.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:22 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:22.925+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:23.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:22 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:22.925+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:23.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:22 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:22.925+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:51.343213+0000 front 2023-07-14T19:25:51.343122+0000 (oldest deadline 2023-07-14T19:26:20.143699+0000) 2023-07-14T19:26:23.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:23 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:23.097+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:23.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:23 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:23.097+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:23.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:23 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:23.097+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:23.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:23 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:23.097+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:23.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:23 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:23.097+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:23.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:23 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:23.097+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:23.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:23 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:23.232+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:23.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:23 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:23.232+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:23.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:23 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:23.232+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:23.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:23 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:23.232+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:23.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:23 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:23.232+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:24.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:23 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:23.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:24.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:23 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:23.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:24.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:23 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:23.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:24.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:23 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:23.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:24.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:23 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:23.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:24.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:23 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:23.916+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:51.343213+0000 front 2023-07-14T19:25:51.343122+0000 (oldest deadline 2023-07-14T19:26:20.143699+0000) 2023-07-14T19:26:24.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:24 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:24.100+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:24.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:24 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:24.100+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:24.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:24 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:24.100+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:24.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:24 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:24.100+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:24.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:24 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:24.100+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:24.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:24 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:24.100+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:24.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:24 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:24.231+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:24.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:24 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:24.231+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:24.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:24 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:24.231+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:24.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:24 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:24.231+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:24.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:24 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:24.231+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:25.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:24 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:24.920+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:25.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:24 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:24.920+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:25.324 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:24 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:24.920+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:25.324 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:24 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:24.920+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:25.325 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:24 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:24.920+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:25.325 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:24 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:24.920+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:51.343213+0000 front 2023-07-14T19:25:51.343122+0000 (oldest deadline 2023-07-14T19:26:20.143699+0000) 2023-07-14T19:26:25.326 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:24 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:24.920+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:26:00.144150+0000 front 2023-07-14T19:26:00.144171+0000 (oldest deadline 2023-07-14T19:26:24.244200+0000) 2023-07-14T19:26:25.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:25 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:25.109+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:25.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:25 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:25.109+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:25.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:25 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:25.109+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:25.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:25 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:25.109+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:25.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:25 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:25.109+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:25.480 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:25 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:25.109+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:25.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:25 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:25.190+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:25.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:25 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:25.190+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:25.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:25 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:25.190+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:25.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:25 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:25.190+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:25.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:25 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:25.190+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:26.321 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:25 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:25.882+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:18.038795+0000 front 2023-07-14T19:25:18.038831+0000 (oldest deadline 2023-07-14T19:25:43.338993+0000) 2023-07-14T19:26:26.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:25 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:25.882+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:23.339568+0000 front 2023-07-14T19:25:23.339686+0000 (oldest deadline 2023-07-14T19:25:47.439497+0000) 2023-07-14T19:26:26.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:25 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:25.882+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.140466+0000 front 2023-07-14T19:25:32.140480+0000 (oldest deadline 2023-07-14T19:25:58.540893+0000) 2023-07-14T19:26:26.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:25 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:25.882+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:40.241681+0000 front 2023-07-14T19:25:40.242016+0000 (oldest deadline 2023-07-14T19:26:03.741888+0000) 2023-07-14T19:26:26.322 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:25 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:25.882+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:46.042883+0000 front 2023-07-14T19:25:46.043001+0000 (oldest deadline 2023-07-14T19:26:11.342745+0000) 2023-07-14T19:26:26.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:25 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:25.882+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:51.343213+0000 front 2023-07-14T19:25:51.343122+0000 (oldest deadline 2023-07-14T19:26:20.143699+0000) 2023-07-14T19:26:26.323 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:25 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9[126108]: 2023-07-14T19:26:25.882+0000 7f5224fdd700 -1 osd.9 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:26:00.144150+0000 front 2023-07-14T19:26:00.144171+0000 (oldest deadline 2023-07-14T19:26:24.244200+0000) 2023-07-14T19:26:26.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:26 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:26.091+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:26.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:26 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:26.091+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:26.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:26 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:26.091+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:26.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:26 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:26.091+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:26.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:26 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:26.091+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:26.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:26 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:26.091+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:26.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:26 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:26.156+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:26.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:26 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:26.156+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:26.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:26 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:26.156+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:26.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:26 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:26.156+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:26.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:26 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:26.156+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:27.071 INFO:journalctl@ceph.osd.9.smithi053.stdout:Jul 14 19:26:26 smithi053 bash[146915]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-9 2023-07-14T19:26:27.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:27 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:27.048+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:27.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:27 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:27.048+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:27.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:27 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:27.048+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:27.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:27 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:27.048+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:27.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:27 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:27.048+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:27.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:27 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:27.048+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:27.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:27 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:27.048+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:26:01.733719+0000 front 2023-07-14T19:26:01.733667+0000 (oldest deadline 2023-07-14T19:26:26.433474+0000) 2023-07-14T19:26:27.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:27 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:27.108+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:27.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:27 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:27.108+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:27.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:27 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:27.108+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:27.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:27 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:27.108+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:27.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:27 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:27.108+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:28.034 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.9.service' 2023-07-14T19:26:28.076 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:26:28.077 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-14T19:26:28.077 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-14T19:26:28.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.10 2023-07-14T19:26:28.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.021+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:28.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.021+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:28.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.021+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:28.480 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.021+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:28.480 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.021+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:28.480 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.021+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:28.481 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.021+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:26:01.733719+0000 front 2023-07-14T19:26:01.733667+0000 (oldest deadline 2023-07-14T19:26:26.433474+0000) 2023-07-14T19:26:28.481 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 systemd[1]: Stopping Ceph osd.10 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:26:28.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:28 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:28.091+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:28.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:28 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:28.091+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:28.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:28 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:28.091+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:28.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:28 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:28.091+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:28.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:28 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:28.091+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:28.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:28 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:28.091+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:28.977 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.486+0000 7f6512b93700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:26:28.978 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.486+0000 7f6512b93700 -1 osd.10 85 *** Got signal Terminated *** 2023-07-14T19:26:28.978 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:28 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:28.486+0000 7f6512b93700 -1 osd.10 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:26:29.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:29 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:29.050+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:29.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:29 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:29.050+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:29.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:29 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:29.050+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:29.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:29 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:29.050+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:29.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:29 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:29.050+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:29.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:29 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:29.050+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:29.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:29 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:29.050+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:26:01.733719+0000 front 2023-07-14T19:26:01.733667+0000 (oldest deadline 2023-07-14T19:26:26.433474+0000) 2023-07-14T19:26:29.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:29 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:29.045+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:29.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:29 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:29.045+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:29.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:29 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:29.045+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:29.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:29 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:29.045+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:29.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:29 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:29.045+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:29.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:29 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:29.045+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:30.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:30.042+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:30.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:30.042+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:30.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:30.042+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:30.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:30.042+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:30.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:30.042+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:30.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:30.042+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:30.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:30 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:30.042+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:26:01.733719+0000 front 2023-07-14T19:26:01.733667+0000 (oldest deadline 2023-07-14T19:26:26.433474+0000) 2023-07-14T19:26:30.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:30 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:30.087+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:30.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:30 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:30.087+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:30.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:30 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:30.087+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:30.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:30 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:30.087+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:30.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:30 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:30.087+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:30.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:30 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:30.087+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:31.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:31.014+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:31.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:31.014+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:31.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:31.014+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:31.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:31.014+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:31.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:31.014+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:31.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:31.014+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:31.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:31 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:31.014+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:26:01.733719+0000 front 2023-07-14T19:26:01.733667+0000 (oldest deadline 2023-07-14T19:26:26.433474+0000) 2023-07-14T19:26:31.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:31 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:31.064+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:31.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:31 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:31.064+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:31.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:31 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:31.064+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:31.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:31 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:31.064+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:31.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:31 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:31.064+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:31.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:31 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:31.064+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:32.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:32 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:32.037+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:32.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:32 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:32.037+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:32.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:32 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:32.037+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:32.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:32 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:32.037+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:32.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:32 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:32.037+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:32.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:32 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:32.037+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:32.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:32 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:32.037+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:26:01.733719+0000 front 2023-07-14T19:26:01.733667+0000 (oldest deadline 2023-07-14T19:26:26.433474+0000) 2023-07-14T19:26:32.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:32 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:32.103+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:32.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:32 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:32.103+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:32.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:32 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:32.103+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:32.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:32 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:32.103+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:32.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:32 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:32.103+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:32.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:32 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:32.103+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:33.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:33 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:33.074+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:16.826600+0000 front 2023-07-14T19:25:16.826550+0000 (oldest deadline 2023-07-14T19:25:40.326647+0000) 2023-07-14T19:26:33.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:33 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:33.074+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:25.027565+0000 front 2023-07-14T19:25:25.027618+0000 (oldest deadline 2023-07-14T19:25:47.327695+0000) 2023-07-14T19:26:33.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:33 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:33.074+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:32.028781+0000 front 2023-07-14T19:25:32.028806+0000 (oldest deadline 2023-07-14T19:25:54.928694+0000) 2023-07-14T19:26:33.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:33 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:33.074+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6814 osd.3 since back 2023-07-14T19:25:37.830253+0000 front 2023-07-14T19:25:37.830230+0000 (oldest deadline 2023-07-14T19:26:01.930115+0000) 2023-07-14T19:26:33.478 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:33 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:33.074+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:47.830932+0000 front 2023-07-14T19:25:47.830976+0000 (oldest deadline 2023-07-14T19:26:13.131138+0000) 2023-07-14T19:26:33.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:33 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:33.074+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.131740+0000 front 2023-07-14T19:25:53.131892+0000 (oldest deadline 2023-07-14T19:26:17.231616+0000) 2023-07-14T19:26:33.479 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:33 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10[119711]: 2023-07-14T19:26:33.074+0000 7f650e188700 -1 osd.10 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:26:01.733719+0000 front 2023-07-14T19:26:01.733667+0000 (oldest deadline 2023-07-14T19:26:26.433474+0000) 2023-07-14T19:26:33.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:33 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:33.096+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:33.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:33 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:33.096+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:33.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:33 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:33.096+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:33.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:33 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:33.096+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:33.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:33 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:33.096+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:33.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:33 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:33.096+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:33.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:33 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:33.096+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2023-07-14T19:26:07.297888+0000 front 2023-07-14T19:26:07.297755+0000 (oldest deadline 2023-07-14T19:26:32.597663+0000) 2023-07-14T19:26:34.477 INFO:journalctl@ceph.osd.10.smithi070.stdout:Jul 14 19:26:34 smithi070 bash[126470]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-10 2023-07-14T19:26:34.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:34 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:34.142+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:34.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:34 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:34.142+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:34.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:34 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:34.142+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:34.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:34 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:34.142+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:34.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:34 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:34.142+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:34.513 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:34 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:34.142+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:34.513 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:34 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:34.142+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2023-07-14T19:26:07.297888+0000 front 2023-07-14T19:26:07.297755+0000 (oldest deadline 2023-07-14T19:26:32.597663+0000) 2023-07-14T19:26:35.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.183+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:35.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.183+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:35.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.183+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:35.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.183+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:35.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.183+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:35.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.183+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:35.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.183+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2023-07-14T19:26:07.297888+0000 front 2023-07-14T19:26:07.297755+0000 (oldest deadline 2023-07-14T19:26:32.597663+0000) 2023-07-14T19:26:35.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.10.service' 2023-07-14T19:26:35.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:26:35.587 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-14T19:26:35.587 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-14T19:26:35.588 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.11 2023-07-14T19:26:35.935 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 systemd[1]: Stopping Ceph osd.11 for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:26:36.189 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.934+0000 7fb7b778a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:26:36.190 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.934+0000 7fb7b778a700 -1 osd.11 85 *** Got signal Terminated *** 2023-07-14T19:26:36.190 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:35 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:35.934+0000 7fb7b778a700 -1 osd.11 85 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:26:36.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:36 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:36.188+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:36.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:36 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:36.188+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:36.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:36 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:36.188+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:36.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:36 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:36.188+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:36.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:36 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:36.188+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:36.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:36 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:36.188+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:36.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:36 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:36.188+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2023-07-14T19:26:07.297888+0000 front 2023-07-14T19:26:07.297755+0000 (oldest deadline 2023-07-14T19:26:32.597663+0000) 2023-07-14T19:26:37.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:37 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:37.163+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:37.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:37 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:37.163+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:37.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:37 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:37.163+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:37.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:37 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:37.163+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:37.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:37 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:37.163+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:37.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:37 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:37.163+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:37.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:37 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:37.163+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2023-07-14T19:26:07.297888+0000 front 2023-07-14T19:26:07.297755+0000 (oldest deadline 2023-07-14T19:26:32.597663+0000) 2023-07-14T19:26:38.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:38 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:38.113+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:38.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:38 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:38.113+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:38.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:38 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:38.113+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:38.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:38 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:38.113+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:38.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:38 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:38.113+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:38.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:38 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:38.113+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:38.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:38 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:38.113+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2023-07-14T19:26:07.297888+0000 front 2023-07-14T19:26:07.297755+0000 (oldest deadline 2023-07-14T19:26:32.597663+0000) 2023-07-14T19:26:39.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:39 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:39.119+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:39.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:39 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:39.119+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:39.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:39 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:39.119+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:39.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:39 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:39.119+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:39.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:39 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:39.119+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:39.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:39 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:39.119+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:39.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:39 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:39.119+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2023-07-14T19:26:07.297888+0000 front 2023-07-14T19:26:07.297755+0000 (oldest deadline 2023-07-14T19:26:32.597663+0000) 2023-07-14T19:26:40.510 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:40 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:40.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6806 osd.0 since back 2023-07-14T19:25:17.289323+0000 front 2023-07-14T19:25:17.289376+0000 (oldest deadline 2023-07-14T19:25:42.589425+0000) 2023-07-14T19:26:40.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:40 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:40.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2023-07-14T19:25:22.589974+0000 front 2023-07-14T19:25:22.589827+0000 (oldest deadline 2023-07-14T19:25:47.290012+0000) 2023-07-14T19:26:40.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:40 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:40.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6804 osd.2 since back 2023-07-14T19:25:33.592110+0000 front 2023-07-14T19:25:33.592148+0000 (oldest deadline 2023-07-14T19:25:58.292321+0000) 2023-07-14T19:26:40.511 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:40 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:40.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2023-07-14T19:25:45.794527+0000 front 2023-07-14T19:25:45.794410+0000 (oldest deadline 2023-07-14T19:26:10.494371+0000) 2023-07-14T19:26:40.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:40 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:40.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.104:6812 osd.5 since back 2023-07-14T19:25:53.795983+0000 front 2023-07-14T19:25:53.796006+0000 (oldest deadline 2023-07-14T19:26:17.896073+0000) 2023-07-14T19:26:40.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:40 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:40.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.53:6822 osd.6 since back 2023-07-14T19:25:57.896527+0000 front 2023-07-14T19:25:57.896510+0000 (oldest deadline 2023-07-14T19:26:27.297158+0000) 2023-07-14T19:26:40.512 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:40 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11[120987]: 2023-07-14T19:26:40.136+0000 7fb7b257e700 -1 osd.11 85 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2023-07-14T19:26:07.297888+0000 front 2023-07-14T19:26:07.297755+0000 (oldest deadline 2023-07-14T19:26:32.597663+0000) 2023-07-14T19:26:41.760 INFO:journalctl@ceph.osd.11.smithi104.stdout:Jul 14 19:26:41 smithi104 bash[128023]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-osd-11 2023-07-14T19:26:42.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@osd.11.service' 2023-07-14T19:26:42.831 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:26:42.831 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-14T19:26:42.832 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-14T19:26:42.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.a 2023-07-14T19:26:43.187 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:26:42 smithi053 systemd[1]: Stopping Ceph mds.a for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:26:43.571 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:26:43 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-a[131375]: 2023-07-14T19:26:43.185+0000 7fb2bb868700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:26:43.572 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:26:43 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-a[131375]: 2023-07-14T19:26:43.185+0000 7fb2bb868700 -1 mds.a *** got signal Terminated *** 2023-07-14T19:26:46.071 INFO:journalctl@ceph.mds.a.smithi053.stdout:Jul 14 19:26:45 smithi053 bash[147183]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-a 2023-07-14T19:26:46.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.a.service' 2023-07-14T19:26:46.194 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:26:46.195 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-14T19:26:46.195 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-14T19:26:46.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.d 2023-07-14T19:26:46.561 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:26:46 smithi053 systemd[1]: Stopping Ceph mds.d for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:26:46.821 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:26:46 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-d[130380]: 2023-07-14T19:26:46.560+0000 7fb1f6e43700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:26:46.822 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:26:46 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-d[130380]: 2023-07-14T19:26:46.560+0000 7fb1f6e43700 -1 mds.d *** got signal Terminated *** 2023-07-14T19:26:50.753 INFO:journalctl@ceph.mds.d.smithi053.stdout:Jul 14 19:26:50 smithi053 bash[147325]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-d 2023-07-14T19:26:50.758 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.d.service' 2023-07-14T19:26:50.793 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:26:50.794 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-14T19:26:50.794 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-14T19:26:50.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.g 2023-07-14T19:26:51.071 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:26:50 smithi053 systemd[1]: Stopping Ceph mds.g for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:26:51.571 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:26:51 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-g[130990]: 2023-07-14T19:26:51.168+0000 7f47e7428700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:26:51.572 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:26:51 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-g[130990]: 2023-07-14T19:26:51.168+0000 7f47e7428700 -1 mds.g *** got signal Terminated *** 2023-07-14T19:26:55.571 INFO:journalctl@ceph.mds.g.smithi053.stdout:Jul 14 19:26:55 smithi053 bash[147468]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-g 2023-07-14T19:26:55.653 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.g.service' 2023-07-14T19:26:55.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:26:55.693 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-14T19:26:55.693 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-14T19:26:55.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.j 2023-07-14T19:26:56.071 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:26:55 smithi053 systemd[1]: Stopping Ceph mds.j for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:26:56.571 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:26:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-j[130001]: 2023-07-14T19:26:56.084+0000 7f1c2f61d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:26:56.572 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:26:56 smithi053 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-j[130001]: 2023-07-14T19:26:56.084+0000 7f1c2f61d700 -1 mds.j *** got signal Terminated *** 2023-07-14T19:27:00.571 INFO:journalctl@ceph.mds.j.smithi053.stdout:Jul 14 19:27:00 smithi053 bash[147616]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-j 2023-07-14T19:27:00.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.j.service' 2023-07-14T19:27:00.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:27:00.622 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-14T19:27:00.623 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-14T19:27:00.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.b 2023-07-14T19:27:00.977 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:27:00 smithi070 systemd[1]: Stopping Ceph mds.b for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:27:01.477 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:27:00 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-b[123413]: 2023-07-14T19:27:00.988+0000 7fd731d38700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:27:01.478 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:27:00 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-b[123413]: 2023-07-14T19:27:00.988+0000 7fd731d38700 -1 mds.b *** got signal Terminated *** 2023-07-14T19:27:05.227 INFO:journalctl@ceph.mds.b.smithi070.stdout:Jul 14 19:27:04 smithi070 bash[126748]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-b 2023-07-14T19:27:05.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.b.service' 2023-07-14T19:27:05.311 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:27:05.312 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-14T19:27:05.313 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-14T19:27:05.313 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.e 2023-07-14T19:27:05.721 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:27:05 smithi070 systemd[1]: Stopping Ceph mds.e for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:27:05.977 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:27:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-e[122724]: 2023-07-14T19:27:05.720+0000 7fe8f80b6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:27:05.978 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:27:05 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-e[122724]: 2023-07-14T19:27:05.720+0000 7fe8f80b6700 -1 mds.e *** got signal Terminated *** 2023-07-14T19:27:09.239 INFO:journalctl@ceph.mds.e.smithi070.stdout:Jul 14 19:27:08 smithi070 bash[126892]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-e 2023-07-14T19:27:09.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.e.service' 2023-07-14T19:27:09.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:27:09.280 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-14T19:27:09.280 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-14T19:27:09.280 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.h 2023-07-14T19:27:09.662 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:27:09 smithi070 systemd[1]: Stopping Ceph mds.h for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:27:09.977 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:27:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-h[123074]: 2023-07-14T19:27:09.660+0000 7fb4a5f56700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:27:09.978 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:27:09 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-h[123074]: 2023-07-14T19:27:09.660+0000 7fb4a5f56700 -1 mds.h *** got signal Terminated *** 2023-07-14T19:27:12.977 INFO:journalctl@ceph.mds.h.smithi070.stdout:Jul 14 19:27:12 smithi070 bash[127032]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-h 2023-07-14T19:27:13.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.h.service' 2023-07-14T19:27:13.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:27:13.245 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-14T19:27:13.245 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-14T19:27:13.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.k 2023-07-14T19:27:13.477 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:27:13 smithi070 systemd[1]: Stopping Ceph mds.k for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:27:13.977 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:27:13 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-k[122384]: 2023-07-14T19:27:13.635+0000 7f5431887700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:27:13.978 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:27:13 smithi070 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-k[122384]: 2023-07-14T19:27:13.635+0000 7f5431887700 -1 mds.k *** got signal Terminated *** 2023-07-14T19:27:18.291 INFO:journalctl@ceph.mds.k.smithi070.stdout:Jul 14 19:27:18 smithi070 bash[127172]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-k 2023-07-14T19:27:18.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.k.service' 2023-07-14T19:27:18.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:27:18.505 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-14T19:27:18.505 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-14T19:27:18.505 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.c 2023-07-14T19:27:18.859 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:27:18 smithi104 systemd[1]: Stopping Ceph mds.c for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:27:19.260 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:27:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-c[124596]: 2023-07-14T19:27:18.858+0000 7f65d4b9a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:27:19.260 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:27:18 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-c[124596]: 2023-07-14T19:27:18.858+0000 7f65d4b9a700 -1 mds.c *** got signal Terminated *** 2023-07-14T19:27:21.010 INFO:journalctl@ceph.mds.c.smithi104.stdout:Jul 14 19:27:20 smithi104 bash[128285]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-c 2023-07-14T19:27:21.104 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.c.service' 2023-07-14T19:27:21.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:27:21.141 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-14T19:27:21.141 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-14T19:27:21.142 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.f 2023-07-14T19:27:21.510 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:27:21 smithi104 systemd[1]: Stopping Ceph mds.f for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:27:22.010 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:27:21 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-f[124252]: 2023-07-14T19:27:21.515+0000 7fe46f64c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:27:22.010 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:27:21 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-f[124252]: 2023-07-14T19:27:21.515+0000 7fe46f64c700 -1 mds.f *** got signal Terminated *** 2023-07-14T19:27:24.987 INFO:journalctl@ceph.mds.f.smithi104.stdout:Jul 14 19:27:24 smithi104 bash[128425]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-f 2023-07-14T19:27:25.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.f.service' 2023-07-14T19:27:25.210 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:27:25.211 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-14T19:27:25.211 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-14T19:27:25.212 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.i 2023-07-14T19:27:25.510 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:27:25 smithi104 systemd[1]: Stopping Ceph mds.i for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:27:26.010 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:27:25 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-i[124944]: 2023-07-14T19:27:25.582+0000 7fb89f7aa700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:27:26.010 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:27:25 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-i[124944]: 2023-07-14T19:27:25.582+0000 7fb89f7aa700 -1 mds.i *** got signal Terminated *** 2023-07-14T19:27:29.260 INFO:journalctl@ceph.mds.i.smithi104.stdout:Jul 14 19:27:29 smithi104 bash[128573]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-i 2023-07-14T19:27:29.337 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.i.service' 2023-07-14T19:27:29.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:27:29.374 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-14T19:27:29.375 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-14T19:27:29.376 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.l 2023-07-14T19:27:29.757 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:27:29 smithi104 systemd[1]: Stopping Ceph mds.l for c5147628-227a-11ee-9b33-001a4aab830c... 2023-07-14T19:27:30.010 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:27:29 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-l[123913]: 2023-07-14T19:27:29.755+0000 7f39f0d15700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:27:30.010 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:27:29 smithi104 ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-l[123913]: 2023-07-14T19:27:29.755+0000 7f39f0d15700 -1 mds.l *** got signal Terminated *** 2023-07-14T19:27:32.260 INFO:journalctl@ceph.mds.l.smithi104.stdout:Jul 14 19:27:31 smithi104 bash[128719]: ceph-c5147628-227a-11ee-9b33-001a4aab830c-mds-l 2023-07-14T19:27:32.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c5147628-227a-11ee-9b33-001a4aab830c@mds.l.service' 2023-07-14T19:27:32.345 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:27:32.346 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-14T19:27:32.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c5147628-227a-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:27:32.664 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:27:34.322 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c5147628-227a-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:27:34.624 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:27:36.185 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c5147628-227a-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:27:36.483 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:27:38.052 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:27:38.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:27:38.126 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:27:38.156 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-14T19:27:38.158 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/remote/smithi053/crash 2023-07-14T19:27:38.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:27:38.205 INFO:teuthology.orchestra.run.smithi053.stderr:tar: /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:27:38.205 INFO:teuthology.orchestra.run.smithi053.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:27:38.207 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/remote/smithi070/crash 2023-07-14T19:27:38.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:27:38.257 INFO:teuthology.orchestra.run.smithi070.stderr:tar: /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:27:38.257 INFO:teuthology.orchestra.run.smithi070.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:27:38.268 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/remote/smithi104/crash 2023-07-14T19:27:38.270 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:27:38.303 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/c5147628-227a-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:27:38.304 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:27:38.305 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-14T19:27:38.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-14T19:27:38.344 INFO:teuthology.orchestra.run.smithi053.stderr:grep: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph.log: No such file or directory 2023-07-14T19:27:38.347 INFO:tasks.cephadm:Compressing logs... 2023-07-14T19:27:38.347 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:27:38.390 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:27:38.392 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:27:38.420 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:27:38.421 INFO:teuthology.orchestra.run.smithi104.stderr:: No such file or directory 2023-07-14T19:27:38.421 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.5.log 2023-07-14T19:27:38.421 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.2.log 2023-07-14T19:27:38.422 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.11.log 2023-07-14T19:27:38.422 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.5.log: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mgr.z.log 2023-07-14T19:27:38.424 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.8.log 2023-07-14T19:27:38.424 INFO:teuthology.orchestra.run.smithi104.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:27:38.424 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.f.log 2023-07-14T19:27:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-14T19:27:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:27:38.426 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.0.log 2023-07-14T19:27:38.427 INFO:teuthology.orchestra.run.smithi053.stderr:gzip /var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.g.log 2023-07-14T19:27:38.427 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.a.log 2023-07-14T19:27:38.427 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.6.log 2023-07-14T19:27:38.427 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.3.log 2023-07-14T19:27:38.427 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.6.log: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.d.log 2023-07-14T19:27:38.428 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.g.log.gz 2023-07-14T19:27:38.428 INFO:teuthology.orchestra.run.smithi053.stderr: 94.6% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.a.log.gz 2023-07-14T19:27:38.428 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mgr.x.log 2023-07-14T19:27:38.428 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.9.log 2023-07-14T19:27:38.428 INFO:teuthology.orchestra.run.smithi070.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:27:38.429 INFO:teuthology.orchestra.run.smithi070.stderr:: No such file or directory 2023-07-14T19:27:38.429 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.10.log 2023-07-14T19:27:38.429 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.h.log 2023-07-14T19:27:38.429 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.e.log 2023-07-14T19:27:38.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.4.log 2023-07-14T19:27:38.430 INFO:teuthology.orchestra.run.smithi070.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:27:38.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.1.log 2023-07-14T19:27:38.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.k.log 2023-07-14T19:27:38.430 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.1.log: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.k.log: 94.6% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.k.log.gz 2023-07-14T19:27:38.430 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:27:38.431 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.7.log 2023-07-14T19:27:38.431 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.8.log: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.f.log: 91.5% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mgr.z.log.gz 2023-07-14T19:27:38.432 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.c.log 2023-07-14T19:27:38.432 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.i.log 2023-07-14T19:27:38.432 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.c.log: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mon.c.log 2023-07-14T19:27:38.433 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-volume.log: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.7.log: 91.5% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:27:38.433 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mgr.x.log: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.9.log: 94.5% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.d.log.gz 2023-07-14T19:27:38.433 INFO:teuthology.orchestra.run.smithi053.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:27:38.434 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:27:38.434 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.j.log 2023-07-14T19:27:38.435 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-volume.log: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.j.log: 94.5% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.j.log.gz 2023-07-14T19:27:38.435 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mon.a.log 2023-07-14T19:27:38.436 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mon.b.log 2023-07-14T19:27:38.436 INFO:teuthology.orchestra.run.smithi070.stderr: 91.3% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.h.log.gz 2023-07-14T19:27:38.436 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.b.log 2023-07-14T19:27:38.436 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mgr.y.log 2023-07-14T19:27:38.437 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mon.a.log: 91.5% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:27:38.437 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mon.c.log: 92.7% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.f.log.gz 2023-07-14T19:27:38.437 INFO:teuthology.orchestra.run.smithi104.stderr: 92.7% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.c.log.gz 2023-07-14T19:27:38.438 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:27:38.438 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.l.log 2023-07-14T19:27:38.439 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.b.log: /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mgr.y.log: 91.3% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.e.log.gz 2023-07-14T19:27:38.440 INFO:teuthology.orchestra.run.smithi070.stderr: 91.3% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mgr.y.log.gz 2023-07-14T19:27:38.442 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.l.log: 94.9% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.l.log.gz 2023-07-14T19:27:38.443 INFO:teuthology.orchestra.run.smithi104.stderr: 91.5% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:27:38.444 INFO:teuthology.orchestra.run.smithi104.stderr: 92.4% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.i.log.gz 2023-07-14T19:27:38.513 INFO:teuthology.orchestra.run.smithi070.stderr: 95.7% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mds.b.log.gz 2023-07-14T19:27:38.522 INFO:teuthology.orchestra.run.smithi104.stderr: 93.4% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.11.log.gz 2023-07-14T19:27:38.542 INFO:teuthology.orchestra.run.smithi053.stderr: 92.9% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.9.log.gz 2023-07-14T19:27:38.555 INFO:teuthology.orchestra.run.smithi104.stderr: 93.0% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.2.log.gz 2023-07-14T19:27:38.555 INFO:teuthology.orchestra.run.smithi104.stderr: 93.0% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.8.log.gz 2023-07-14T19:27:38.556 INFO:teuthology.orchestra.run.smithi070.stderr: 93.0% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.7.log.gz 2023-07-14T19:27:38.564 INFO:teuthology.orchestra.run.smithi053.stderr: 93.0% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.3.log.gz 2023-07-14T19:27:38.572 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mgr.x.log.gz 2023-07-14T19:27:38.584 INFO:teuthology.orchestra.run.smithi053.stderr: 92.8% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.6.log.gz 2023-07-14T19:27:38.588 INFO:teuthology.orchestra.run.smithi070.stderr: 92.9% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.4.log.gz 2023-07-14T19:27:38.591 INFO:teuthology.orchestra.run.smithi053.stderr: 93.2% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.0.log.gz 2023-07-14T19:27:38.592 INFO:teuthology.orchestra.run.smithi104.stderr: 92.9% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.5.log.gz 2023-07-14T19:27:38.597 INFO:teuthology.orchestra.run.smithi104.stderr: 92.0% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mon.c.log.gz 2023-07-14T19:27:38.599 INFO:teuthology.orchestra.run.smithi104.stderr: 2023-07-14T19:27:38.599 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m0.190s 2023-07-14T19:27:38.599 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m0.740s 2023-07-14T19:27:38.599 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.052s 2023-07-14T19:27:38.600 INFO:teuthology.orchestra.run.smithi070.stderr: 93.1% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.10.log.gz 2023-07-14T19:27:38.611 INFO:teuthology.orchestra.run.smithi070.stderr: 92.0% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mon.b.log.gz 2023-07-14T19:27:38.625 INFO:teuthology.orchestra.run.smithi070.stderr: 93.3% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-osd.1.log.gz 2023-07-14T19:27:38.626 INFO:teuthology.orchestra.run.smithi070.stderr: 2023-07-14T19:27:38.626 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m0.219s 2023-07-14T19:27:38.627 INFO:teuthology.orchestra.run.smithi070.stderr:user 0m0.950s 2023-07-14T19:27:38.627 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m0.057s 2023-07-14T19:27:38.856 INFO:teuthology.orchestra.run.smithi053.stderr: 90.9% -- replaced with /var/log/ceph/c5147628-227a-11ee-9b33-001a4aab830c/ceph-mon.a.log.gz 2023-07-14T19:27:38.858 INFO:teuthology.orchestra.run.smithi053.stderr: 2023-07-14T19:27:38.858 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m0.452s 2023-07-14T19:27:38.858 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m1.163s 2023-07-14T19:27:38.858 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.053s 2023-07-14T19:27:38.859 INFO:tasks.cephadm:Archiving logs... 2023-07-14T19:27:38.859 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/remote/smithi053/log 2023-07-14T19:27:38.860 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:27:39.056 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/remote/smithi070/log 2023-07-14T19:27:39.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:27:39.174 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/remote/smithi104/log 2023-07-14T19:27:39.175 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:27:39.287 INFO:tasks.cephadm:Removing cluster... 2023-07-14T19:27:39.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c5147628-227a-11ee-9b33-001a4aab830c --force 2023-07-14T19:27:39.580 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:27:40.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c5147628-227a-11ee-9b33-001a4aab830c --force 2023-07-14T19:27:40.335 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:27:40.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c5147628-227a-11ee-9b33-001a4aab830c --force 2023-07-14T19:27:41.093 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: c5147628-227a-11ee-9b33-001a4aab830c 2023-07-14T19:27:41.548 INFO:tasks.cephadm:Removing cephadm ... 2023-07-14T19:27:41.549 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:27:41.572 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:27:41.592 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:27:41.610 INFO:tasks.cephadm:Teardown complete 2023-07-14T19:27:41.611 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-14T19:27:41.628 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-14T19:27:41.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:27:41.633 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:27:41.636 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:27:45.251 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-14T19:27:45.262 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-14T19:27:45.262 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:27:45.265 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:27:45.268 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:27:45.282 INFO:teuthology.orchestra.run.smithi053.stderr:bash: ntpq: command not found 2023-07-14T19:27:45.284 INFO:teuthology.orchestra.run.smithi070.stderr:bash: ntpq: command not found 2023-07-14T19:27:45.287 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:27:45.287 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2023-07-14T19:27:45.287 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +422us[ +422us] +/- 93ms 2023-07-14T19:27:45.288 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 18 -4763us[-4763us] +/- 87ms 2023-07-14T19:27:45.288 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv03.front.sepia.ceph.com 3 6 377 20 +3993us[+3984us] +/- 82ms 2023-07-14T19:27:45.288 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:27:45.289 INFO:teuthology.orchestra.run.smithi104.stderr:bash: ntpq: command not found 2023-07-14T19:27:45.290 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:27:45.290 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2023-07-14T19:27:45.291 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 +462us[ +465us] +/- 93ms 2023-07-14T19:27:45.291 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -4736us[-4736us] +/- 87ms 2023-07-14T19:27:45.291 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv03.front.sepia.ceph.com 3 6 377 20 +4010us[+4013us] +/- 82ms 2023-07-14T19:27:45.291 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:27:45.292 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:27:45.292 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2023-07-14T19:27:45.293 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 23 +734us[ +704us] +/- 93ms 2023-07-14T19:27:45.293 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 84 -4443us[-4487us] +/- 86ms 2023-07-14T19:27:45.293 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv03.front.sepia.ceph.com 3 6 377 19 +4266us[+4236us] +/- 82ms 2023-07-14T19:27:45.294 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:27:45.294 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-14T19:27:45.303 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-14T19:27:45.305 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-14T19:27:45.315 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:27:45.436 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:27:45.535 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:27:45.635 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:27:45.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:27:45.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:27:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:27:45.706 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:27:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:27:45.740 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-14T19:27:45.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-14T19:27:45.767 INFO:teuthology.task.internal:Duration was 1925.420314 seconds 2023-07-14T19:27:45.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-14T19:27:45.780 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-14T19:27:45.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:27:45.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:27:45.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:27:45.831 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:27:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:27:45.834 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:27:46.271 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-14T19:27:46.272 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2023-07-14T19:27:46.272 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:27:46.297 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2023-07-14T19:27:46.297 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:27:46.323 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2023-07-14T19:27:46.324 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:27:46.349 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-14T19:27:46.350 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:27:46.355 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:27:46.367 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:27:46.440 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-14T19:27:46.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:27:46.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:27:46.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:27:46.548 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-14T19:27:46.559 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-14T19:27:46.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:27:46.586 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:27:46.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:27:46.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-14T19:27:46.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:27:46.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:27:46.658 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:27:46.681 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2023-07-14T19:27:46.682 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2023-07-14T19:27:46.708 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2023-07-14T19:27:46.730 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:27:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:27:46.751 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:27:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:27:46.767 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:27:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:27:46.787 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-14T19:27:46.798 INFO:teuthology.task.internal:Transferring archived files... 2023-07-14T19:27:46.799 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/remote/smithi053 2023-07-14T19:27:46.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:27:46.838 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/remote/smithi070 2023-07-14T19:27:46.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:27:46.875 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051/remote/smithi104 2023-07-14T19:27:46.875 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:27:46.910 INFO:teuthology.task.internal:Removing archive directory... 2023-07-14T19:27:46.911 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:27:46.913 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:27:46.916 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:27:46.962 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-14T19:27:46.975 INFO:teuthology.task.internal:Not uploading archives. 2023-07-14T19:27:46.975 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-14T19:27:46.985 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-14T19:27:46.986 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:27:46.988 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:27:46.990 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:27:47.002 INFO:teuthology.orchestra.run.smithi053.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:27 /home/ubuntu/cephtest 2023-07-14T19:27:47.004 INFO:teuthology.orchestra.run.smithi070.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:27 /home/ubuntu/cephtest 2023-07-14T19:27:47.019 INFO:teuthology.orchestra.run.smithi104.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:27 /home/ubuntu/cephtest 2023-07-14T19:27:47.020 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-14T19:27:47.030 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-14T19:27:47.065 INFO:teuthology.nuke:Checking targets against current locks 2023-07-14T19:27:52.101 DEBUG:teuthology.nuke:shortname: smithi053 2023-07-14T19:27:52.102 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:27:52.125 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:49:07.309276', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T19:27:57.135 DEBUG:teuthology.nuke:shortname: smithi070 2023-07-14T19:27:57.135 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:28:07.171 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:49:07.307930', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T19:28:12.179 DEBUG:teuthology.nuke:shortname: smithi104 2023-07-14T19:28:12.180 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:28:12.217 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338051', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:49:07.306370', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T19:28:12.228 INFO:teuthology.orchestra.console:Power off smithi053 2023-07-14T19:28:12.229 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:28:12.248 INFO:teuthology.orchestra.console:Power off smithi070 2023-07-14T19:28:12.249 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:28:12.260 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:12.279 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:12.320 INFO:teuthology.orchestra.console:Power off smithi104 2023-07-14T19:28:12.321 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:28:12.345 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:16.294 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:16.304 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:16.368 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:21.409 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:21.419 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:21.480 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:21.693 INFO:teuthology.orchestra.console:Power off for smithi104 completed 2023-07-14T19:28:26.521 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:26.532 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:26.734 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2023-07-14T19:28:31.648 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:36.760 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:41.876 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:28:42.090 INFO:teuthology.orchestra.console:Power off for smithi070 completed 2023-07-14T19:28:42.197 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} duration: 1925.420313835144 failure_reason: 'Command failed on smithi053 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs _nogroup distributed 1''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9d92ce77e42b460cb34a1053e5b0e77e status: fail success: false 2023-07-14T19:28:42.197 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T19:28:43.416 INFO:teuthology.run:FAIL